Data processing system and method, communication system and method, and charging device and method

ABSTRACT

In a communication system for serving data corresponding to a request from the user and charging the user for the data served to the user, when the user designates a result of a desired one of a plurality of different operations, a usage management unit selects a one, corresponding to the user&#39;s designation, of a plurality of processors which effect operations corresponding to a plurality of different algorithms, respectively. When the processor corresponding to the designation is selected, an algorithm processor makes an operation corresponding to the designation. A management center charges the user for the use of the algorithm.

TECHNICAL FIELD

[0001] The present invention generally relates to a data processingapparatus and method for paying an amount charged for data served to auser, an charging management apparatus and method for charging a userfor data served to the user, a communication system and method, acommunication apparatus and method included in the communication system,and a medium having stored therein a program which allows an informationprocessing apparatus to charge a user for data served to the user.

BACKGROUND ART

[0002] For example, in a data service system including a data server orprovider and users to whom data are served from the data server, theuser is charged proportionally to an amount of data served to the useror on a flat-rate basis not depending upon the served data amount, forexample.

[0003] Also, there is disclosed in the U.S. Pat. No. 6,041,316 a networksystem in which low-quality data are normally served but the user can beserved with higher-quality data with having to pay an additional chargefor the higher quality.

[0004] Also, the Japanese Published Unexamined Application No. 164552 of1998 discloses a video-on-demand transmitter and terminal which make itpossible to serve a video program whose quality meets the user's demandand charge the user on a program-quality basis. In the video-on-demandtransmitter and terminal disclosed in this Japanese Published UnexaminedApplication, there is used a data processing algorithm such as MPEG-1 orMPEG-2 (these are similar to each other) upon which the quality of aserved video program depends and thus the charge for a served videoprogram is varied depending upon whether or not the served video programis a one compressed with MPEG-1 or MPEG-2. For example, MPEG-1 is usedat the transmitter while MPEG-2 is used at the terminal.

[0005] Namely, in the above Japanese Published Unexamined Application,the charge to the user is varied depending upon the function of theapplication layer in the video-on-demand transmitter and terminal suchas the signal processing technique used for data served to the user.

[0006] The above transmitter and terminal has a function torecord/reproduce a hardware control layer function such as power supplyto the internal circuit in them, but there is not disclosed any systemwith which the user can selectively grade up the functions of thehardware control layer.

[0007] With the above technique, however, a signal processing techniquedesignated by the user once will be fixed and the quality of videoprogram and signal processing will not be selectable even if the stateof transmission is changed. Namely, any optimum service will not beavailable to the user.

DISCLOSURE OF THE INVENTION

[0008] Accordingly, the present invention has an object to overcome theabove-mentioned drawbacks of the prior art by providing a dataprocessing apparatus and method, charging management apparatus andmethod, communication system and method, communication apparatus andmethod, and storage medium having stored therein a program executed inthe data processing apparatus etc., enabling a data service and chargingadaptive to different demands of the user, more particularly, a chargingadaptive to a profit of the user as well as to the value of data servedby the data server to the user.

[0009] The above object can be attained by providing a data processingsystem including:

[0010] a designation input unit capable of designating desired methodsfor at least an application layer and hardware control layer,respectively;

[0011] a signal processor capable of selectively carrying out methodsrelated to the application layer;

[0012] a hardware controller capable of selectively carrying out methodsrelated to the hardware control layer; and

[0013] a controller to control the signal processor or hardwarecontroller based on the designation from the designation input unit tocarry out the designated method.

[0014] In the above data processing system, the signal processorincludes a storage unit to store a plurality of methods, and thecontroller reads, from the storage unit, one method in response to adesignation from the designation input unit and carries out the method.

[0015] The above data processing system further includes a transmitterto send data, a receiver to receive transmitted data, and an output unitto output received content data. The signal processor is controlled bythe controller to process supplied content data by carrying out a methodselected on the basis of the designation supplied from the designationinput unit, and output the processed content data to the transmitter.

[0016] Also, the above object can be attained by providing acommunication system with a charging management apparatus, including:

[0017] a processor including:

[0018] a designation input unit capable of designating desired methodsfor at least an application layer and hardware control layer,respectively;

[0019] a request signal transmitter to send a signal of request for amethod designated by the designation input unit;

[0020] a receiver to receive a reply to the request signal;

[0021] a signal processor capable of selectively carrying out methodsrelated to the application layer;

[0022] a hardware controller capable of selectively carrying out methodsrelated to the hardware control layer; and

[0023] a controller to control the signal processor or hardwarecontroller to carry out a method designated by the designation inputunit;

[0024] a charging management apparatus including:

[0025] a request signal receiver to receive the request signal;

[0026] a judgment unit to judge, based on the received request signal,whether the method is available not;

[0027] a judgment result transmitter to send the judgment result fromthe judgment unit to the receiver;

[0028] a usage-charge storage unit to store charge information on theuse of each of a plurality of methods stored in the signal processor orhardware controller; and

[0029] a computing unit for computing, based on information stored inthe usage-charge storage unit, information indicative of an amountcharged to the user having made designation to the designation inputunit, and outputting the charged amount information.

[0030] Also, the above object can be attained by providing a chargingmanagement apparatus including:

[0031] a request signal receiver to receive, from an external apparatus,a signal of request for permission to use desired methods related to anapplication layer and hardware control layer, respectively, of theexternal apparatus;

[0032] a judgment unit to judge, based on the permission request signalreceived by the request signal receiver, whether the method is availablenot;

[0033] a judgment result transmitter to send the judgment result fromthe judgment unit to the receiver;

[0034] a usage-charge storage unit to store charge information on theuse of each of a plurality of methods; and

[0035] a computing unit for computing, based on information stored inthe usage-charge storage unit, information indicative of an amountcharged to the user having made designation to the designation inputunit, and outputting the charged amount information.

[0036] In the above charging management apparatus, the charged amountinformation computing unit computes, based on information stored in theusage-charge storage unit, charged amount information by summing acharge for use of an automatic selection method for automaticallyselecting one of a plurality of methods and a charge for the use of amethod corresponding to a compression algorithm selected by theautomatic selection method.

[0037] Also, the above object can be attained by providing acommunication system including first and second communicationapparatuses and a charging apparatus,

[0038] the first communication apparatus including:

[0039] a designation input unit capable of designating desired methodsfor at least an application layer and hardware control layer,respectively, of a transmitter;

[0040] a request signal transmitter to send a signal of request for amethod designated by the designation input unit;

[0041] a receiver to receive content data processed by the method madeavailable by the request signal transmitter;

[0042] a hardware controller capable of selectively carrying out methodsrelated to the hardware control layer; and

[0043] a first controller to control the hardware controller to carryout a method designated by the designation input unit;

[0044] the second communication apparatus including:

[0045] a signal processor for selectively carrying out methods relatedto the application layer to process input content data;

[0046] a second controller for controlling the method designated by thedesignation input unit to be carried out based on the request signalsent from the request signal transmitter; and

[0047] a transmitter to send the processed content data to outside; and

[0048] the charging management apparatus including:

[0049] a usage-charge storage unit to store charge information on theuse of each of a plurality of methods stored in the signal processor orhardware controller; and

[0050] a computing unit for computing, based on information stored inthe usage-charge storage unit, information indicative of an amountcharged to the user having made designation to the designation inputunit, and outputting the charged amount information.

[0051] In the above communication system, the communication apparatus isto make communications of content data, and includes a receiver toreceive an external signal of request for carrying out a method relatedto the application layer, a signal processor for selectively carryingout methods related to the application layer to process input contentdata, a controller to control the signal processor to carry out adesignated method, and a transmitter to send the processed content datato outside.

[0052] Also, the above object can be attained by providing acommunication method for communication of content data and a program forcontrolling a communication system in which the communication method iscarried out, by which desired methods are designated for at least theapplication layer of the transmitter and the hardware control layer ofthe receiver, respectively, a signal of request for the designatedmethod is sent, the signal processor processes content data supplied tothe transmitter by carrying out the method according to the sent requestsignal, and the processed content data are sent from the transmitter tothe receiver. The receiver receives the sent content data, the methodcarried out by the hardware controller controls power supply to theinternal circuit of the receiver, information indicative of an amountcharged to the user having designated the method is computed based oninformation stored in the usage-charge storage unit having storedtherein information on a charge for the use of each of a plurality ofmethods stored in the signal processor or hardware controller, and thecharged amount information is outputted.

[0053] These objects and other objects, features and advantages of thepresent invention will become more apparent from the following detaileddescription of the best mode for carrying out the present invention whentaken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0054]FIG. 1 is a block diagram of the communication system according tothe present invention.

[0055]FIG. 2 is a block diagram of the transmitter included in thecommunication system in FIG. 1.

[0056]FIG. 3 shows a flow of operations made in the transmitter in FIG.2.

[0057]FIG. 4 is a block diagram of the receiver included in thecommunication system in FIG. 1.

[0058]FIG. 5 shows a flow of operations made in the receiver in FIG. 4.

[0059]FIG. 6 is a block diagram of the transmission processor includedin the transmitter in FIG. 2.

[0060]FIG. 7 is a block diagram of the encoder of the transmissionprocessor in FIG. 6.

[0061]FIGS. 8A, 8B and 8C explain the hierarchical encoding/decoding.

[0062]FIG. 9 shows a flow of operations made in the transmissionprocessor in FIG. 6.

[0063]FIG. 10 is a block diagram of the reception processor included inthe receiver in FIG. 4.

[0064]FIG. 11 is a block diagram of the decoder included in thereception processor in FIG. 10.

[0065]FIG. 12 is a block diagram of the synthesizer of the receiver inFIG. 4.

[0066]FIG. 13 shows a flow of operations made in the synthesizer in FIG.12.

[0067]FIGS. 14A, 14B and 14C show examples of displays of an image onthe image output unit of the receiver in FIG. 4.

[0068]FIGS. 15A and 15B explain the relation between the spatialresolution and time resolution of an image sent from the transmitter andreceiver, included in the communication system in FIG. 1.

[0069]FIG. 16 is a block diagram of the essential portion of thecontroller included in the transmission processor in FIG. 6, intendedfor prediction of a focus of interest.

[0070]FIG. 17 explains an example of a feature reflecting a tendency ofuser's interest.

[0071]FIG. 18 shows a flow of operations made in the controller includedin the transmission processor in FIG. 6, intended for prediction of afocus of interest.

[0072]FIG. 19 is a block diagram of a model of charging effected betweena processor having methods pre-installed therein and the managementcenter.

[0073]FIG. 20 is a block diagram of a charging model intended foracquisition of a method from the management center.

[0074]FIG. 21 is a block diagram of the essential portion of thetransmitter and receiver, intended for an actual data communicationsystem having the charging model applied therein.

[0075]FIG. 22 shows a flow of operations made in the transmitter in FIG.21, intended for the actual data communication system having thecharging model applied therein.

[0076]FIG. 23 shows a flow of operations made in the receiver in FIG.21, intended for the actual data communication system having thecharging model applied therein.

[0077]FIG. 24 shows a flow of operations made in the management centerin FIG. 21, intended for the actual data communication system having thecharging model applied therein.

[0078]FIG. 25 shows a flow of operations made according to theoptimization algorithm intended for automatic selection of an algorithm.

[0079]FIGS. 26A and 26B explain an object extracted from an image andits motion.

[0080]FIG. 27 shows a flow of operations made in the objectencoding-compression.

[0081]FIGS. 28A and 28B explain description of an image with atime-space model.

[0082]FIG. 29 is a block diagram of an example of the data communicationsystem intended for a learning to make to determine a coefficient of thepredictive coding circuit for classification-adaptive predictive coding.

[0083]FIG. 30 is a block diagram of an example of the predictive codingcircuit for classification-adaptive predictive coding.

[0084]FIG. 31 is a block diagram of another example of the predictivecoding circuit for classification-adaptive predictive coding.

[0085]FIG. 32 is a block diagram of an example of the classificationcircuit for classification-adaptive predictive coding.

[0086]FIG. 33 is a block diagram of an example of the decoder forclassification-adaptive predictive coding.

[0087]FIG. 34 explains the concept of an operation per process (perclock) in automatic control of the power consumption.

[0088]FIG. 35 shows a flow of operations made according to the automaticpower-consumption control algorithm to judge whether a power control isto be done or not adaptively to a process.

[0089]FIG. 36 shows a flow of operations made in performance of thepower control function.

[0090]FIG. 37 is a block diagram of the computer according to thepresent invention.

BEST MODE FOR CARRYING OUT THE INVENTION

[0091] In the PCT Published Unexamined Application No. WO01/11889A1 ofthe Applicant of the present invention, there is disclosed an imageprocessor which effects the spatial resolution control disclosed in theJapanese Published Unexamined Application No. 112856 of 1998 as well asthe time resolution control in order to control the amount ofinformation in an area designated by clicking a mouse or the like at thedata receiving side.

[0092] Further, the above PCT Published Unexamined Application No.WO01/11889A1 discloses an algorithm by which an object the user isinterested in is extracted from a position clicked with the mouse by theuser at the data receiving side and tine interval between clicks.Moreover, it discloses a technique by which a user's preference islearned from a feature of the object the user has been interested in andan object the user is interested in is automatically predicted, tothereby improve the image quality of the object.

[0093] The preferred modes of carrying out the present invention will bedescribed herebelow with reference to the accompanying drawings.

[0094] The data communication system according to the present inventionis configured as shown in FIG. 1. The term “system” used herein refersto a logical assembly of a plurality of apparatuses whether theapparatuses of different constructions are in the same enclosure or not.

[0095] As shown in FIG. 1, the data communication system includes atleast two terminals 1 and 2 each being for example a mobile telephone,PHS (personal handy-phone system: registered trademark) or the like, aradio base station 3 or 5 which makes radio communications with theterminal 1 or 2, and an exchange 4 such as a telephone station whichprovides a connection between the base stations 3 and 5. Note that theradio base stations 3 and 5 are identical to each other or differentfrom each other. With the above system configuration, the terminals 1and 2 can send signals from one to the other of them and receive signalsfrom their counterpart, via a transmission line formed from the radiobase stations 3 and 5, exchange 4, etc.

[0096] There will be described in detail later the operations made forcharging at each time of data service and data reception, such asoperations made for charging for data service to the terminals 1 and 2via the management center 103 in FIG. 1 and for data transmission andreception between the terminals 1 and 2 in the data communication systemaccording to the present invention and the configuration of the system.

[0097] In the data communication system shown in FIG. 1, each of theterminals 1 and 2 being a mobile telephone, PHS or the like includes atleast a key pad 8 for input of a phone number, characters, symbols,etc., a microphone 10 for input of a sound, a speaker 9 for output of asound, a video camera 6 having an imaging device and optical system,capable of picking up a still image and moving image, respectively, anda display unit 7 capable of displaying characters and symbols as well asan image.

[0098] Between the terminals 1 and 2, there can be transferred soundsignals as well as image data captured by the video camera 6. Therefore,each of the terminals 1 and 2 can display an image picked up by itscounterpart on the display unit 7.

[0099] There will be described here by way of example a datacommunication in which for example the terminal 1 sends image data whilethe terminal 2 receives the image data from the terminal 1. In thefollowing description, the terminal 1 or 2 will be referred to as“transmitter 1” or “receiver 2”, respectively, wherever appropriate.

[0100] In this case, image data are sent along with information on itsframe rate from the transmitter 1 to tile receiver 2 via a transmissionline including the base stations 3 and 5, exchange 4, etc. Receiving theimage data sent from the transmitter 1, the receiver 2 will display, onthe display unit 7 such as a liquid crystal display (LCD) or the like, amoving image reproduced from the image data according the received framerate information. On the other hand, the receiver 2 will send, to thetransmitter 1 via the transmission line, control information used tocontrol the spatial resolution and time resolution of the imagedisplayed on the display unit 7. That is, the receiver 2 will send, tothe transmitter 1, control information (“click data” as designation datawhich will further be described later) used at the transmitter 1 tolocalize an area the user of the receiver 2 is interested in.

[0101] Receiving the control information (click data) from the receiver2, the transmitter 1 will localize, based on the click data, the imagearea (interesting area) the user of the receiver 2 is interested in froman image to be displayed at the receiver 2, such as an image picked upby the video camera 6 of the transmitter 1. Further, the transmitter 1will control the amount of information in image data to be sent to thereceiver 2 in such a manner that the spatial resolution and timeresolution of the localized image area will be changed while satisfyinggiven conditions. Note that in case each of the transmitter 1 andreceiver 2 is a PHS terminal for example, the frequency range of thetransmission line will be 1895.1500 to 1905.9500 MHz and thetransmission rate will be 128 kbps (bit per second).

[0102] Referring now to FIG. 2, there is illustrated in the form of ablock diagram the transmitter 1 included in the data communicationsystem shown in FIG. 1. As shown, the transmitter 1 further includes animage input unit 11 consisting of the video camera 6 having for examplethe imaging device (CCD: charge-coupled device) and optical system, animage signal processing circuit to generate image data from the imagesignal picked up by the video camera 6, etc. That is, the user of thetransmitter 1 images a desired object by the video camera 6, the imagesignal processing circuit generates image data from the image suppliedfrom the video camera 6, and the image input unit 11 sends the imagedata to a pre-processor 12.

[0103] The pre-processor 12 includes mainly a background extraction unit13, object extraction unit 14 and an appended information computationcircuit 15. Further detailed description of the pre-processor 12 will begiven later.

[0104] Based on the click data sent from the receiver 2, the objectextraction unit 14 of the pre-processor 12 extracts, from the imagecaptured by the video camera 6 of the image input unit 11, an area theuser of the receiver 2 is interested in, that is, an interesting area,and supplies a transmission processor 16 with image data correspondingto the extracted interesting area. Note that in case the image picked upby the video camera 6 of the image input unit 11 includes a plurality ofareas the user of the receiver 2 is interested in, the object extractionunit 14 will supply image data corresponding to the plurality ofinteresting areas to the transmission processor 16. Also, the image datacorresponding to the interesting area extracted by the object extractionunit 14 is also supplied to the appended information computation unit15.

[0105] The area the user is interested in is for example an object suchas a substance found in an image. Note that the “object” referred toherein is one of pieces of an image divided by a unit and which can beprocessed per unit and for processing each of special substances in animage, the substance is defined as “object”. According to the presentinvention, object data are extracted from an image on the basis of clickdata and processed per object. Note that object is prepared in differentmanners depending upon the content of a required image.

[0106] In the object extraction unit 14, an object as an example ofinteresting area (will be referred to as “appropriate object” hereunder)is extracted as will be described below. Note that the interesting areahas not to always be an object but may be an image area other than anobject, image area in an object or background image part which will bedescribed later or the like. The present invention will be describedconcerning an object as an interesting area.

[0107] In the embodiment of the present invention, small-object imagedata corresponding to click data derived from clicking of an image bythe user are extracted and the small-object image data are connected ornot to each other to extract object image data and then extract anobject image from the object image data.

[0108] Also, when the interest of the user of the receiver 2 is directedto another area, the object extraction unit 14 will detect the change ofinterest to extract an object image being a new interesting area basedon the result of detection of the change in interest of the user. Theobject extraction effected in the object extraction unit 14, that is,the localization of an interesting area and detection of an area towhich the user's interest has shifted will further be described later.

[0109] Next, the background extraction unit 13 of the pre-processor 12extracts, based on the object extraction result supplied from the objectextraction unit 14, signals (will be referred to as “background imagedata” hereunder) equivalent to a background portion (image area otherthan the interesting area; will be referred to as “background image”hereunder) of an image from the image data supplied from the image inputunit 11, and supplies the extracted background image data to thetransmission processor 16 and appended information computation unit 15.The background image is a flat image area whose activity is low andhaving no special meaning as an image. Of course, the background imageincludes an image having no special meaning as well as an object theuser is interested in, but for the simplicity of the explanation, theabove flat image area will be taken as a background image in thefollowing description of the present invention.

[0110] The appended information computation unit 15 detects, based onthe background image data supplied from the background extraction unit13, a movement of the background caused by the change in imagingdirection of the image input unit 11 during imaging, for example abackground movement vector indicative of a panning and tilting. Theappended information computation unit 15 detects, based on the imagedata of an object (will be referred to as “object image data” hereunder)supplied from the object extraction unit 14, an object movement vectorindicative of a movement of the object. The appended informationcomputation unit 15 supplies the transmission processor 16 with themovement vectors as appended information. Also, based on the objectimage data supplied from the object extraction unit 14, the appendedinformation computation unit 15 supplies the transmission processor 16with an image picked up by the video camera 6 of the image input unit11, that is, information on the object such as a position, profileindicating a shape, etc. of the object in a frame image, as appendedinformation. Namely, the object extraction unit 14 extracts alsoinformation oil the object such as position, shape, etc. of the objectduring extraction of an object image, and supplies them to the appendedinformation computation unit 15. The appended information computationunit 15 will output the information on the object as appendedinformation.

[0111] Based on the click data supplied from the receiver 2, thetransmission processor 16 encodes the object image data from the objectextraction unit 14, background image data from the background extractionunit 13 and appended information from the appended informationcomputation unit 15 so as to meet the requirement for a data rate atwhich data can be transmitted on the transmission line while raising thespatial and time resolutions of the object image in an image to bedisplayed on the display unit 2. Further, the transmission processor 16multiplexes the encoded object image data (will be referred to as“object encoded data” hereunder), background image data (will bereferred to as “background encoded data” hereunder) and appendedinformation (will be referred to as “appended information encoded data”hereunder), and sends the multiplexed data along with frame rateinformation to the receiver 2 via the transmission line.

[0112] Next, the operations of the transmitter 1 shown in FIG. 2 will bedescribed with reference to the flow chart shown in FIG. 3.

[0113] First in step S1 in FIG. 3, the video camera 6 of the image inputunit 11 at the transmitter 1 captures an image, and sends the image datato the pre-processor 12.

[0114] Next in step S2, the transmitter 1 receives the click data sentfrom the transmitter 2, and supplies the click data to the pre-processor12.

[0115] In step S3, the pre-processor 12 having received the image dataand click data pre-processes the data for background extraction, objectextraction and appended information computation, and sends backgroundimage data, object image data and appended information thus obtained tothe transmission processor 16. The object extraction includes detectionof interesting-object change as well.

[0116] In step S4, the transmission processor 16 computes amounts of theobject image data, background image data and appended information so asto meet the requirement for a data rate at which data can be transmittedvia the transmission line, and multiplexes the object image data,background image data and appended information according to their dataamount by encoding them as will further be described later. Thereafter,the transmission processor 16 will send the multiplexed data along withframe rate information to the receiver 2 via the transmission line.

[0117] Subsequently, the procedure returns to step S1 and similaroperations are repeated.

[0118]FIG. 4 shows the construction of the receiver 2 included in thedata communication system shown in FIG. 1.

[0119] The multiplexed data sent from the transmitter 1 via thetransmission line are received by a reception processor 21 in thereceiver 2 shown in FIG. 4. The reception processor 21 will demultiplex,from the received multiplexed-data, the background encoded data, objectencoded data and appended information encoded data and decode them, andsend the decoded background image data, object image data and appendedinformation to a synthesizer 22.

[0120] The synthesizer 22 will synthesize an image from the decodedbackground image date, object image data and appended informationsupplied from the reception processor 21, and supply the synthesizedimage signals to an image output unit 23. Also, the synthesizer 22controls the spatial and time resolutions of the synthesized image basedon the click data supplied from a click data input unit 24.

[0121] Based on the supplied image data, the image output unit 23 willgenerate a drive signal for driving the liquid crystal display or thelike in the display unit 7 and send the drive signal to the liquidcrystal display or the like at a frame rate which is based on the framerate information received along with the aforementioned multiplexeddata. Thus, the image synthesized by the synthesizer 22 is displayed onthe display unit 7.

[0122] When the user operates the key pad 8 having the function as apointing device to designate a coordinate position of an image on thedisplay unit 7, the click data input unit 24 generates click dataindicating a clicked position corresponding to the operation of the keypad 8 by the user, that is, a coordinate position, and a clicked time.Namely, when the user clicks the key pad 8 to designate an interestingarea being a desired image portion in an image displayed on the displayunit 7, the click data input unit 24 generates click data indicatingcoordinate information on the clicked position and a clicked time. Theclick data generated by the click data input unit 24 are sent to thesynthesizer 22 and click data transmission unit 25.

[0123] Receiving the click data from the click data input unit 24, theclick data transmission unit 25 sends it to the transmitter 1 via thetransmission line.

[0124] Next, the operations of the receiver 2 shown in FIG. 4 will beoutlined with reference to the flow chart shown in FIG. 5.

[0125] First in step S11 in FIG. 5, the reception processor 21 in thereceiver 2 receives the multiplexed data from the transmitter 1 via thetransmission line.

[0126] Next in step S12, the reception processor 21 demultiplexes, fromthe multiplexed data, the background encoded data, object encoded dataand appended information encoded data, and then decodes the thusdemultiplexed encoded data. The decoded background image data, objectimage data and appended information are sent to the synthesizer 22.

[0127] In step S13, the click data input unit 24 of the receiver 2acquires the click data by clicking the key pad 8 by the user, sends itto the synthesizer 22 and also to the click data transmission unit 25,and the click data are sent from the click data transmission unit 25 tothe transmitter 1.

[0128] Next in step S14, the synthesizer 22 synthesizes an image fromthe background image data, object image data and appended informationsupplied from the reception processor 21 and the click data suppliedfrom the click data input unit 24, and controls the spatial and timeresolutions of the synthesized image.

[0129] Thereafter in step S15, the image output unit 23 has the liquidcrystal display or the like in the display unit 7 display the imagesynthesized by the synthesizer 22 thereon on the basis of the frame rateinformation received along with the multiplexed data.

[0130] Thereafter, the procedure returns to step S11, and similaroperations are repeated.

[0131] Next, the construction of the transmission processor 16 includedin the transmitter 1 shown in FIG. 2 will be described in detail withreference to FIG. 6.

[0132] As shown in FIG. 6, the transmission processor 16 is suppliedwith the background image data, object image data and appendedinformation from the pre-processor 12 in FIG. 2. The background imagedata, object image data and appended information are supplied to anencoder 31 and controller 35.

[0133] The encoder 31 hierarchically encodes the supplied backgroundimage data, object image data and appended information as will furtherbe described later, and supplies each data thus encoded to a multiplexer(MUX) 32.

[0134] Under the control of the controller 35, the MUX 32 selects thebackground encoded data, object encoded data and appended informationencoded data supplied from the encoder 31 and supplies them asmultiplexed data to a transmission unit 33.

[0135] The transmission unit 33 modulates the multiplexed data suppliedfrom the MUX 32 according to the aforementioned frame rate informationand the transmission standard for a downstream transmission line, andsends the modulated multiplexed data to the receiver 2 via thedownstream transmission line.

[0136] Also, the data amount computation unit 34 monitors themultiplexed data outputted from the MUX 32 to the transmission unit 33,compute a data rate for the multiplexed data and supplies the computeddata rate to the controller 35.

[0137] The controller 35 controls the output of multiplexed data fromthe MUX 32 so that the data rate computed by the data amount computationunit 34 will not exceed the transmission rate of the transmission linewhile receiving the click data received from the receiver 2 via thetransmission line to control the multiplexing of the encoded data in theMUX 32.

[0138] The encoder 31 shown in FIG. 6 is constructed as shown in detailin FIG. 7.

[0139] In the encoder 31 in FIG. 7, the background image data aresupplied to a difference computation unit 41B. The differencecomputation unit 41B subtracts one frame-precedent, already processedbackground image data from background image data contained in an imageframe (will be referred to as “current frame” hereunder whereverappropriate) supplied from a local decoder 44B and going to be processedat present, and supplies difference data of the background image (willbe referred to as “background image difference data” hereunder) as thesubtraction result to a hierarchical encoder 42B.

[0140] The hierarchical encoder 42B hierarchically encodes thebackground image difference data from the difference computation unit41B, and supplies data obtained by the encoding, that is, backgroundencoded data, to a storage unit 43B.

[0141] The storage unit 43B provisionally stores the background encodeddata supplied from the hierarchical encoder 42B. The background encodeddata stored in the storage unit 43B are sent to the MUX 32 in FIG. 6.

[0142] Further, the background encoded data stored in the storage unit43B are supplied to the local decoder 44B. The local decoder 44B locallydecodes the background encoded data to decode their initial backgroundimage data, and supplies the decoded background image data to thedifference computation unit 41B. The background image data thus decodedby the local decoder 44B are used in the difference computation unit 41Bto acquire data on a difference from background image data in a nextframe.

[0143] In the encoder 31 in FIG. 7, the object image data are suppliedto a difference computation unit 41F. The difference computation unit41F subtracts one frame-preceding, already processed background imagedata from object image data contained in an image frame (current frame)supplied from a local decoder 44F and going to be processed at present,and supplies difference data of the object (will be referred to as“object image difference data” hereunder) as the subtraction result to ahierarchical encoder 42F.

[0144] The hierarchical encoder 42F hierarchically encodes thebackground image difference data from the difference computation unit41F, and supplies data obtained by the encoding (object encoded data) toa storage unit 43F.

[0145] The storage unit 43F provisionally stores the object encoded datasupplied from the hierarchical encoder 42F. The object encoded datastored in the storage unit 43F are sent to the MUX 32 in FIG. 6.

[0146] Further, the object encoded data stored in the storage unit 43Fare supplied to the local decoder 44F. The local decoder 44F locallydecodes the object encoded data to decode their initial object imagedata, and supplies the decoded object image data to the differencecomputation unit 41F. The object image data thus decoded by the localdecoder 44F are used in the difference computation unit 41F to acquiredata on a difference from object image data in a next frame.

[0147] Note that in case there exists a plurality of objects (#1, #2,#3, . . . ), image data corresponding to the plurality of objects aresubject to difference computation, hierarchical encoding, storage andlocal decoding by the difference computation unit 41 F, hierarchicalencoder 42F, storage unit 43F and local decoder 44F, respectively.

[0148] Also, in the encoder 31 shown in FIG. 7, the appended informationis supplied to a VLC (variable-length encoder) 45. The VLC 45 makesvariable-length encoding of the appended information. Thevariable-length encoding method may be a one which can compress the databy reducing the redundance. Namely, the variable-length encoding may bethe run-length encoding, Huffman encoding or the like. Thevariable-length encoded appended information is sent as theaforementioned appended information encoded, data to the MUX 32 in FIG.6.

[0149] Next, the hierarchical encoding effected in the encoder 31 inFIG. 7 and the decoding effected at the receiving side correspondinglyto the hierarchical encoding will be described with reference to FIG. 8.

[0150] The encoder 31 in FIG. 7 makes, for each of 3 layers, forexample, such a hierarchical encoding as to take a mean value (meanpixel value) of 4 pixels consisting of 2 horizontal pixels and 2vertical pixels in a low layer for example as a value of one pixel in alayer one level higher than the low layer. Note that the term “pixelvalue” used herein refers to a difference obtained by the differencecomputation effected as a preliminary operation for the hierarchicalencoding, namely, a difference of each pixel. Of course, in case nodifference computation is done before the hierarchical encoding, thepixel value is not such a specified one.

[0151] The above will further be described below on the assumption thatan image in the lowest layer (first layer) consists of 4 horizontalpixels and 4 vertical pixels (will be referred to as “4×4 pixels”hereunder) for example as shown in FIG. 8A. In this case, thehierarchical encoding will be such that there is calculated a mean valueof four pixels h00, h01, h02 and h03 including the two horizontal pixelsand two vertical pixels (will be referred to as “2×2 pixels” hereunder)at the upper left of the 4×4 pixels and the mean value is taken as thevalue of a pixel m0 at the upper left in the second layer. Similarly, amean value of 2×2 pixels h10, h11, h12 and h13 at the upper right of the4×4 pixels in the first layer is taken as the value of a pixel m1 at theupper right of the second layer; mean value of 2×2 pixels h20, h21, h22and h23 at the upper left of the 4×4 pixels in the first layer is takenas the value of a pixel m2 at the lower left of the second layer; andmean value of 2×2 pixels h30, h31, h32 and h33 at the lower right of the4×4 pixels in the first layer is taken as the value of a pixel m3 at thelower right of the second layer. Further in the hierarchical encoding, amean value of four pixels m0, m1, m2 and m3 included in the 2×2 pixelsin the second layer is determined and taken as the value of a pixel q inthe third layer (highest layer).

[0152] In the encoder 31 in FIG. 7, the hierarchical encoding iseffected as having been described just above. Note that with such ahierarchical encoding, the spatial resolution of an image in the highestlayer (third layer) is lowest, that of images in the lower layers willbe higher and that of an image in the lowest layer (first layer) will behighest.

[0153] In case all the above pixels h00 to h03, h10 to h13, h20 to h23,h30 to h33, m0 to m3 and q are to be sent, the data amount will belarger by the pixels m0 to m3 in the second layer and pixel q in thethird layer than when only the pixels in the lowest layer are sent.

[0154] To decrease the data amount for sending, the pixel q in the thirdlayer is embedded in place of for example the pixel m3, at the upperright, of the pixels m0 to m3 in the second layer as shown in FIG. 8Band the data of the second layer thus consisting of the pixels m0, m1,m2 and q and data of the first layer are sent. Thus, the data amount canbe smaller by the data amount of the third layer.

[0155] For sending the data in an amount decreased more than in themeasure shown in FIG. 8B, the pixel m0 in the second layer is replacedby for example the pixel h03, at the lower right, of the 2×2 pixels h00to h03 in the first layer, used to determine the pixel m0, similarly thepixel ml in the second layer is replaced by for example the pixel h13,at the lower right, of the 2×2 pixels h10 to h13 in the first layer,used to determine the pixel m1, also the m2 in the second layer isreplaced by for example the pixel h23, at the lower right, of the 2×2pixels h20 to h23 in the first layer, used to determine the pixel m2, asshown in FIG. 8C, and further the q in the third layer, buried in anpixel at the lower right of the pixels m0 to m3 in the second layer inFIG. 8B is replaced by for example the pixel h33, at the lower right, ofthe 2×2 pixels h30 to h33 in the first layer. The data amount in thethird and second layers can thus be reduced. That is, in the exampleshown in FIG. 8C, 16 pixels (4×4 pixels) are thus transmitted, whichnumber of pixels is the same as that of the pixels in the lowest (first)layer as shown in FIG. 8A. Therefore, it is possible in this case tosend data equivalent to pixels in each of the first to third layers inan amount not increased.

[0156] Note that the pixel m3 in the second layer, replaced with thepixel q as in FIG. 8B, and the pixels h03, h13, h23 and h33 in the firstlayer, replaced with the pixels m0, m1, m2 and q, as in FIG. 8C, can bedecoded as will be described below.

[0157] Namely, since the value of the pixel q is a mean value of thepixels m0 to m3, it can be given by an equation of q=(m0+m1+m2+m3)/4.Thus, an equation of m3=4×q−(m0+m1+m2) can be used to determine (decode)the value of the pixel m3 in the second layer from the pixel q in thethird layer and pixels m0 to m2 in the second layer.

[0158] Also, since the value of the pixel m0 is a mean value of thepixels h00 to h03, it can be given by an equation ofm0−(h00+h01+h02+h03)/4. Thus, an equation of h03=4×m0−(h00+h01+h02) canbe used to determine the value of the pixel h03 in the first layer fromthe pixel m0 in the second layer and pixels h00 to h02 in the firstlayer. Similarly, the value of each of the pixels h13, h23 and h33 canbe determined.

[0159] As in the above, pixels included in a layer and not sent can bedecoded from sent pixels included in the layer and sent pixels includedin a one-level higher layer.

[0160] Next, the transmission processing effected in the transmissionprocessor 16 in FIG. 6 will be described with reference to the flowchart in FIG. 9.

[0161] First in step S21, the controller 35 in the transmissionprocessor 16 judges whether click data has been sent from the receiver2. If it judges that no click data has been transmitted from thereceiver 2, namely, if the controller 35 has not received any clickdata, the controller 35 will control, in step S22, the MUX 32 to selectand multiplex background encoded data, object encoded data and appendedinformation encoded data so that the receiver 2 can display an imagewith an ordinary time resolution such as a default time resolution.

[0162] That is, when for example 30 frames/sec is set as the ordinarytime resolution, the receiver 2 will display an image at a rate of 30frames/sec. In this case, when multiplexed data has been sent at thetransmission rate of the transmission line while maintaining the timeresolution of 30 frames/sec, the MUX 32 will select and multiplex theencoded data of background, object and appended information so that thespatial resolution of an image displayed at the receiver 2 will behighest.

[0163] More particularly, in case the hierarchical encoding has beeneffected with the three layers for example as in the above, if only thedata in the third layer can be sent at the transmission rate of thetransmission line for display of an image at the rate of 30 frames/sec,the MUX 32 will select the encoded data of background, object andappended data for display of the image data in the third layer. In thiscase, the receiver 2 ill display an image at the time resolution is 30frames/sec and at horizontal and vertical spatial resolutions being aquarter of those of the first layer image data as their initial data.

[0164] Next in step S23, the transmission processor 16 will send, fromthe transmission unit 33, the multiplexed data from the MUX 32 alongwith the aforementioned set frame rate information via the transmissionline, and then the procedure returns to step S21.

[0165] If the controller 35 judges in step S21 that click data has beensent from the receiver 2, namely, if it has received the click data, itwill recognize, in step S24, based on the click data, a clicked positionbeing a coordinate of a focus of interest designated by the user byoperating the clock data input unit 24 of the receiver 2, and a clickedtime.

[0166] Then in step S25, the controller 35 will localize, based on thecoordinate of the focus of interest, and clicked time, an area the userof the receiver 2 is interested in and set the thus localizedinteresting area as a preferred range in which the spatial resolution ofan image displayed at the receiver 2 is preferentially improved, tothereby detect an image in the preferred range and correspondingappended information. Note that in the present invention, the imageinside the preferred range corresponds to an object image while an imageoutside the preferred range corresponds to an image outside theinteresting area such as a background image.

[0167] In step S26, the controller 35 controls the MUX 32 to select andmultiplex encoded data of an image inside the preferred range (objectimage), image outside the preferred range (background image) andappended information so that the image inside the preferred range willbe displayed with a higher spatial resolution at the receiver 2. Namely,when having received clock data from the receiver 2, the controller 35will control the MUX 32 to improve the spatial resolution of an imageinside the preferred range by the sacrifice of the time resolution.

[0168] In the above, the controller 35 controls the MUX 32 to improvethe spatial resolution of an image inside the preferred range. However,the controller 35 may control the MUX 32 to improve the time resolutionof an image inside the preferred range, that is, the frame rate of animage inside the preferred range. In this case, the controller 35 mayattain the improvement by the sacrifice of the spatial resolution. Also,the controller 35 may control the total information amount by handlingan image corresponding to a background image outside the preferred rangeas a still image.

[0169] Thus, for an image inside the preferred range, the MUX 32preferentially selects and multiplexes encoded data for display of imagedata in the third layer and also in the second layer, and outputs themultiplexed data.

[0170] Further, in step S26, the controller 35 controls the MUX 32 toinsert information on the position, size, etc. of the preferred rangeinto appended information selected as the multiplexed data, and thengoes to step S23.

[0171] In step S23, the transmission unit 33 sends the multiplexed dataoutput from the MUX 32 along with frame rate information via thetransmission line, and then the procedure returns to step S21.

[0172] For the simplicity of the explanation, it is assumed here that instep S26, encoded data for display of image data in the third layer arecontinuously selected for an image outside the preferred range, forexample, a background image, as in step S22. In this case, in thecontroller 35, the amount of the multiplexed data in step S26 will belarger by an image with a higher spatial resolution inside the preferredrange, that is, image data in the second layer for the object image,than in step S22.

[0173] At this time, even if it is intended to display an image at arate of for example 30 frames/sec, since the transmission rate of thetransmission line permits to display only the image data in the thirdlayer, the multiplexed data including the data in the second layer,acquired in step S26, will not be such data as permits to display animage at the rate of 30 frames/sec.

[0174] In this case, the transmission unit 33 will send multiplexed datawhose rate is lower than 30 frames/sec or is 0 frame/sec in an extremecase, namely, a still image. Thus, at the receiver 2, there will bedisplayed an image included in the preferred range and whose horizontaland vertical spatial resolutions are a half of those of their initialimage (image in the first layer), that is, an image (image in the secondlayer) whose horizontal and vertical spatial resolutions are doublethose of a third layer image having so far been displayed. At this time,however, the time resolution of an image displayed at the receiver 2will be less than 30 frames/sec.

[0175] After data in the second layer for an image inside the preferredrange are sent as in the above, if it is judged in step S21 that clockdata has been sent from the receiver 2 as in the above, namely, when theuser continuously operates the clock data input unit 24 to designate afocus of interest identical to or near the preceding one, the focus ofinterest identical to or near the preceding one is recognized in stepS24, and the same preferred range as the preceding one is set in stepS25, and the procedure goes to step S26. Thereby, in step S26, thecontroller 35 will control the MUX 32 to select and multiplex encodeddata so that an image inside the preferred range can be displayed with ahigher spatial resolution at the receiver 2.

[0176] Since the encoded data of images in the third and second layersand information appended to the images are preferentially selected forthe image inside the preferred range as in the above, the encoded dataof images in the first layer and information appended to the images arealso preferentially selected and multiplexed. The high-resolutioninformation is inserted into the appended information as having beendescribed with respect to step S26, and the multiplexed data from theMUX 32 are sent along with the frame rate information from thetransmission unit 33 via the transmission line in step S23, and then theprocedure returns to step S21.

[0177] In this case, the receiver 2 will display an image included inthe preferred range and whose spatial resolution is the same as that oftheir initial image (image in the first layer), that is, an image (imagein the first layer) whose horizontal and vertical spatial resolutionsare 4 times higher than those of a fourth layer image displayed first.However, an image whose time resolution is lower than the 30 frames/secor is 0 frame/sec in an extreme case is handled as a still image.

[0178] With the above operations, since data intended for improvement ofthe spatial resolution of an image inside the preferred range includinga focus of interest, namely, an interesting area such as an objectimage, are preferentially sent if the user of the receiver 2continuously operates the click data input unit 24 to designate forexample the same focus of interest, that is, an interesting area, thespatial resolution of the image inside the preferred range including thefocus of interest is gradually improved with the result that the imagein the preferred range will be displayed more definitely. That is, aninteresting area being an image of a portion the user of the receiver 2is interested in, for example, an object image, will be displayed moredefinitely.

[0179] As in the above, since image data sending is controlled so thatthe spatial resolution or time resolution of an interesting area beingan image within a preferred area localized with a focus of interestwhich is based on click data, such as an object image, is changed withina resolution range corresponding to the transmission rate of atransmission line, an image corresponding to a focus of interest, sentat a limited transmission rate, can be displayed at the receiver 2 witha higher spatial resolution. That is, by improving the spatialresolution of an object image inside a preferred range at the sacrificeof the time resolution of the image, the object image, even when sent atthe limited transmission rate, can be displayed at the receiver 2 moredefinitely, namely, with a higher spatial resolution.

[0180] Next, the reception processor 21 included in the receiver 2 shownin FIG. 4 will be described in further detail below with reference toFIG. 10.

[0181] As shown in FIG. 10, the multiplexed data supplied via thetransmission line are received and modulated by a reception unit 51, andthen supplied to a demultiplexer (DMUX) 52.

[0182] The DMUX 52 demultiplexed the multiplexed data supplied from thereception unit 51 into background encoded data, object encoded data andappended information encoded data, and supplies these encoded data to adecoder 53.

[0183] Reversely following the encoding-compression procedure, thedecoder 53 decodes the encoded data of background, object or appendedinformation (data derived from encoding of the difference in thisembodiment) to their respective initial data, and outputs it to thesynthesizer 22 shown in FIG. 4.

[0184]FIG. 11 shows in detail the decoder 53 in FIG. 10.

[0185] As shown in FIG. 11, the hierarchically encoded background imagedifference data being background encoded data are supplied to an adder61B which is also supplied with background image data preceding by oneframe the background encoded data, stored in a storage unit 62B andalready decoded. The adder 61B adds, to the supplied background imagedifference data, the background image data one frame before thedifference data, supplied from the storage unit 62B, to thereby decodebackground image data in a layer required for the current frame. Thedecoded background image data are supplied to and stored in the storageunit 62B, and then read and supplied to the adder 61B while being sentto the synthesizer 22 in FIG. 4.

[0186] The hierarchically encoded object image difference data beingobject encoded data are supplied to an adder 61F which is also suppliedwith object image data one frame before the difference data, stored in astorage unit 62F and already decoded. By adding, to the supplied objectimage difference data, object image data one frame before the differencedata, supplied from th storage unit 62F, the adder 61F decodes objectimage data in a layer required for the current frame. The decoded objectimage data are supplied to and stored in the storage unit 62F, and thenread and supplied to the adder 61F while being sent to the synthesizer22 in FIG. 4. Note that when there is a plurality of objects, the adder61F and storage unit 62F will repeat the above-mentioned hierarchicaldecoding for the difference data of the plurality of objects.

[0187] The above-mentioned variable length-encoded appended informationbeing appended information encoded data are supplied to a reverse VLC(variable-length coder) unit 63 where they are variable length-decodedto their initial appended information. The initial appended informationis supplied to the synthesizer 22.

[0188] Note that the local decoder 44B in FIG. 7 is constructedsimilarly to the adder 61B and storage unit 62B and the local decoder44F is also constructed similarly to the adder 61 F and storage unit62F.

[0189] Next, the synthesizer 22 included in the receiver 2 shown in FIG.4 will be described in detail below with reference to FIG. 12.

[0190] As shown in FIG. 12, from the decoder 53 in FIG. 10, backgroundimage data are supplied to a background write unit 71, object image dataare supplied to a object write unit 72, and appended information issupplied to a background write unit 71, object write unit 72 and asynthesizer 77.

[0191] The background write unit 71 writes the supplied background imagedata one after another to the background memory 73. In case there is forexample a movement of the background, due to a panning or tilting duringimaging by the video camera 6 in the transmitter 1, the background writeunit 71 will write the background image data to the background memory 73with the background being positioned based on a background movementvector included in the appended information. Therefore, the backgroundmemory 73 can store data on an image spatially wider than one frame ofimage.

[0192] The object write nit 72 will write the supplied object image dataone after another to the object memory 75. Note that in case there isfor example a plurality of objects, the object write unit 72 will writeimage data of the plurality of objects to the object memory 75 for eachobject. Also, for write of image data of objects having the same objectnumber which will further be described later, namely, same object data,the object write unit 72 will write, to the object memory 75, new objectimage data, namely, object image data newly supplied to the object writememory 72 in place of object image data already stored in the objectmemory 75.

[0193] Further, when an object whose spatial resolution is high havebeen written to the object memory 75, the object write unit 72 willchange, from “0” to “1”, the object flag stored at an address in anobject flag memory 76 correspondingly to each of pixels forming theobject in consideration. More specifically, when writing object imagedata to the object memory 75, the object write unit 72 will makereference to the object flag memory 76. No object image data whosespatial resolution is low will be written to the object memory 75 havingalready stored therein an object whose flag is “1”, namely, image dataof an object whose spatial resolution is high. Therefore, basically,each time object image data are supplied to the object write unit 72,they are written to the object memory 75; however, no object image datawhose spatial resolution is low will be written to the object memory 75having already stored therein object image data whose spatial resolutionis high. As a result, in the object memory 75, each time object imagedata whose spatial resolution is high are supplied to the object writeunit 72, the number of object images whose spatial resolution is highwill be larger.

[0194] The synthesizer 77 reads a background image of a current frame tobe displayed at the present from the background image data stored in thebackground memory 73 based on the background movement vector included inthe appended information while pasting, to the background image, theobject image stored in the object memory 75 based on the object movementvector included in the appended information, whereby an image in thecurrent frame is reproduced and supplied to a display memory 78.

[0195] Further, upon reception of click data from the click data inputunit 24 in FIG. 4, the synthesizer 77 reads, from the object memory 75,object image data containing a coordinate position of a focus ofinterest included in the click data, and supplies the data to the subwindow memory 79.

[0196] There is further provided a display memory 78 as a so-called VRAM(video read-only memory) to buffer or provisionally store an image inthe current frame supplied from the synthesizer 77 and then read it outfor supply to the image output unit 23 in FIG. 4. Also, the sub windowmemory 79 provisionally stores object image data supplied from thesynthesizer 77 and then reads it for supply to the image output unit 23in FIG. 4. At this time, the display unit 7 driven by the image outputunit 23 will display, along with the image in the current frame, a subwindow which will further be described later, and display an objectimage in the sub window.

[0197] Next, the operations effected in the synthesizer 22 in FIG. 12will be described below with reference to the flow chart in FIG. 13.

[0198] First in step S31, the object write unit 72 writes the objectimage data supplied from the decoder 53 in FIG. 10 as in the above onthe basis of an object flag stored in the object flag memory 75.

[0199] More particularly, referring to the object flag stored in theobject flag memory 76, the object write unit 72 writes object image datasupplied thereto at an address, in the object memory 75, correspondingto a pixel for which the object flag is “0”, and only object image datasupplied thereto and whose spatial resolution is high at an address, inthe object memory 75, corresponding to a pixel for which the object flagis “1”.

[0200] Note that when object image data are written at an address, inthe object memory 75, where object image data are already stored, theywill be written over the existing object image data in the object memory75.

[0201] Thereafter in step S32, the object write unit 72 judges whetherthe appended information includes high-resolution information. If it isjudged in step S32 that the appended information includeshigh-resolution, namely, when click data are sent to the transmitter 1by operating the clock data input unit 24 in FIG. 4 by the user of thereceiver 2 and thus object image data whose spatial resolution is highare sent for an image inside the preferred range from the transmitter 1,the object write unit 72 goes to step S33 where it will set apredetermined object flag in the object flag memory 76 to “1”.

[0202] That is, when object image data whose spatial resolution is highare sent for the image inside the preferred range from the transmitter1, they are written to the object memory 75 in step S3 1. Thus in stepS33, the object flag for pixels included in an object image whosespatial resolution is high is set to “1”.

[0203] Thereafter the procedure goes to step S34 where the synthesizer77 will read object image data inside the preferred range from theobject memory 75, and write them to the sub window memory 79.

[0204] Namely, when it is judged in step S32 that the appendedinformation includes high-resolution, click data have been sent to thetransmitter 1 by operating the clock data input unit 24 in FIG. 4 by theuser of the receiver 2 and thus object image data whose spatialresolution is high have been sent for an image inside the preferredrange from the transmitter 1. The click data supplied to the transmitter1 are also supplied to the synthesizer 77. Upon reception of the clickdata, the synthesizer 77 will recognize, in step S34, the preferredrange from the coordinate of the focus of interest an clicked timeincluded in the click data, read an object, sent from the transmitter 1,included in the preferred range and having a high spatial resolution,from the object memory 75, and write the data to the sub window memory79.

[0205] Then in step S35, the synthesizer 77 reads, based on thebackground movement vector included in the appended information,background image data in the current frame from the background imagedata stored in the background memory 73, reads object image data to bedisplayed in the current frame from the object memory 75, and furthercombines the background image data in the current frame and object imagedata read from the object memory 75 according to the object movementvector included in the appended information. Thus, the synthesizer 77reproduces the image in the current frame and writes it to the displaymemory 78. That is, the synthesizer 77 writes the background image datato the display memory 78 for example, and then writes the object imagedata over the background image data, thereby writing, to the displaymemory 78, the image data in the current frame obtained by combining thebackground image and object image.

[0206] As in the above, the image data in the current frame written tothe display memory 78, and object image data written to the sub windowmemory 79 will be supplied to the image output unit 23 in FIG. 4 anddisplayed on the display unit 7.

[0207] On the other hand, if it is judged in step S32 that the appendedinformation includes no high-resolution information, namely, when theclick data input unit 24 has not been operated by the user of thereceiver 2, the procedure skips over steps S33 and S34 to step S35 wherethe synthesizer 77 will read the background image data in the currentframe from the background memory 73 and necessary object image data fromthe object memory 75, to combine the background image in the currentframe and object image read from the object memory 75 according to theappended information. Thus, the synthesizer 77 reproduces image data inthe current frame and writes them to the display memory 78. Then theprocedure returns to step S3 1 and similar operations will be repeated.

[0208] With the above operations for the synthesis, when the user of thereceiver 2 has not operated the click data input unit 24, that is, whenno clicking has been made at the click data input unit 24, an imagewhose spatial resolution is low will be displayed on the display screenof the display unit 7 with a default time resolution as shown in FIG.14A. Note that FIG. 14A shows an example in which an object image whosespatial resolution is low is being moved rightward over a backgroundimage whose spatial resolution is low.

[0209] When the user of the receiver 2 moves the cursor over the objectimage by operating the click data input unit 24 and clicks with thecursor on the object image, click data is sent to the transmitter 1 andthe transmitter 1 receives data intended for display, as a highspatial-resolution image, of an image inside a preferred range localizedbased on the click data by the sacrifice of the time resolution. As theresult, there will be displayed on the display screen of the displayunit 7, as shown in FIG. 14B, an image corresponding to an object imageincluded in the preferred range around the clicked position and whosetime resolution is low but whose spatial resolution is graduallyimproved. That is, an image is displayed which corresponds to an objectimage included in the preferred range and whose spatial resolution isgradually improved correspondingly to a time when clicking has been madeon the Image.

[0210] Further, on the display unit 7, the sub window is opened and animage corresponding to an object in an extracted preferred rangeincluding a clicked position is displayed in the sub window with thespatial resolution of the object being gradually improved, as shown inFIG. 14B.

[0211] Thereafter, when the user of the receiver 2 stops clicking withthe click data input unit 24, the synthesizer 77 reads background imagedata in the current frame from the background memory 73 and object imagedata from the object memory 75, combines the background image data andobject image data according to the appended data, and writes the data tothe display memory 78, in step S35 as having been described above. As inthe above, since the object image data whose spatial resolution has beenelevated by clicking is continuously stored as it is in the objectmemory 75, the object image whose spatial resolution has thus beenimproved by clicking is moved according to the appended informationmovement vector is displayed in a due position in the current frame onthe display unit 7 as shown in FIG. 14C.

[0212] Therefore, by clicking in a position where an object image whosedetail is to be observed, the user of the receiver 2 will be able toview an object image having an improved spatial resolution. Thus, theuser will be able to view a detailed image of an object.

[0213] Note that since the background image data is stored in thebackground memory 73 as in the above, the transmitter 1 has not to sendany background sent once and whose spatial resolution is low. Therefore,the transmission rate for the background can be allocated preferentiallyto sending of object image data whose spatial resolution is higher.

[0214] In the above case, the object image data having the spatialresolution raised by clicking is stored in the object memory 75 and theobject image having the high spatial resolution is pasted on thebackground image after the clicking is stopped. Thus, the object imagedisplayed at the receiver 2 will have a high spatial resolution but willnot reflect any change in state of an object image picked up at thetransmitter 1.

[0215] So, with the object flag being disregarded after the clicking isstopped, the object image data stored in the storage unit 62F in thedecoder 53 shown in FIG. 11 can be written over the object image datastored in the object memory 75 and whose spatial resolution is high.That is, since object image data sent from the transmitter 1 are storedone after another into the storage unit 62F of the decoder 53, theobject image in an image displayed on the display unit 7 will be made toreflect a change in state of the object captured at the transmitter 1 asin the above by writing the object image data to the object memory 75.However, the displayed object image will have a low spatial resolution.

[0216] Next, the relation between the spatial and time resolutions of animage sent from the transmitter 1 to the receiver 2 via the transmissionline will be described below with reference to FIG. 15.

[0217] Assume here that the transmission rate of the transmission lineis R [bps] and a background image and data including three objects #1 to#3 are sent from the transmitter 1. For the simplicity of theexplanation, no consideration will be given to the appended information,and it is assumed that for displaying the background image and objectimages #1 to #3 with a certain spatial resolution, the same data amountis required for each of the images.

[0218] In this case, when no clicking has been made at the receiver 2,the transmitter 1 will send the background image and object images #1 to#3 each at a rate R/4 [bps] being a quarter of the transmission rate ofthe transmission line, as shown in FIG. 15A. Note that when the ordinarytime resolution is 1/T frame/sec, the transmitter 1 will send one frameof each of the background image and object images #1 to #3 in a maximumof T sec. Therefore in this case, there will be displayed on thereceiver 2 background image and object images #1 to #3 each having aspatial resolution of T×R/4 bits/frame.

[0219] When the user clicks at a time t, and in a position of the objectimage #1 for example, the transmitter 1 will stop sending the backgroundimage and object images #2 and #3 for example while sending only theobject image #1 at the full transmission rate R of the transmissionline, as shown in FIG. 15A. Thereafter, when the user stops clicking ata time t₂ which is later by a time 4T than the time t₁, the transmitter1 will send the background image and object images #1 to #3 again at atransmission rate of R/4.

[0220] Therefore, while the user is clicking, 4T×R bits of the object #1are sent. So, when the time resolution during clicking is 0 frame/sec,the receiver 2 will display the object image #1 with a spatialresolution of 4T×R bits/frame. That is, when the horizontal and verticalspatial resolutions have been improved to the same extent, the timeresolution at the receiver 2 is 0 frame/sec but the object image #1clicked by the user will be displayed with horizontal and verticalspatial resolutions 4 times higher than those before clicking (={squareroot}{square root over ( )}4T×WR(T×R/4 bits))).

[0221] Thus, the spatial resolution can be improved at the sacrifice ofthe time resolution, and the spatial resolution of object image the useris interested in can be improved more rapidly than when the timeresolution is sacrificed.

[0222] In an example shown in FIG. 15A, while the object image #1 isbeing clicked, a transmission rate of 0 frame/sec is set for sending thebackground image and other object images #2 and #3 so that these datawill not be sent. In an example shown in FIG. 15B, however, a hightransmission rate may be allocated for sending the object image #1 whilea low transmission rate may be allocated for sending the backgroundimage and object images #2 and #3.

[0223] Even if the clicking is made, the transmission rate allocated forsending the background image and object images #1 to #3 can be kept asR/4. That is, since the spatial resolution is improved at the sacrificeof the time resolution, sending of the data takes a time even withoutallocating any other transmission rate.

[0224] In the above, object image having the spatial resolution elevatedby clicking is stored in the object memory 75, and after the clicking isstopped, the object image having the high spatial resolution is pastedon the background image. However, where the high spatial-resolutionobject image is to be pasted on the background image depends upon anobject movement vector included in the appended information on theobject, sent from the transmitter 1 later.

[0225] Therefore, since the receiver 2 has to recognize an object imagein a frame, corresponding to an image in a frame adjacent to the formerframe, the object extraction unit 14 of the transmitter 1 appendsinformation intended for use by the receiver 2 to make such arecognition for extraction of an object.

[0226] In the above, there has been described an example in which when afocus of interest is designated by operating the click data input unit24 by the user of the receiver 2, data sending is controlled in thetransmitter 1 to improve the spatial resolution of an image inside apreferred range including the focus of interest at the sacrifice of thetime resolution of the image. In addition, the transmitter 1 can learnfor example the preference of the user of the receiver 2, andpredictively detects, based on the result of learning, an object etc.which, it is considered, the user of the receiver 2 desires to displaywith a high spatial resolution, to thereby control data sending so thatthe object can be displayed with a high spatial resolution.

[0227]FIG. 16 shows, by way of example, the construction of thecontroller 35 in FIG. 6, intended for such a control of data sending.

[0228] As shown in FIG. 16, the controller 35 includes a preferred rangesetting unit 91, selection controller 92 and a feature extraction unit93. The preferred range setting unit 91 receives click data sent fromthe receiver 2 to set a preferred range as having previously beendescribed, and the set preferred range is supplied to the selectioncontroller 92 and feature extraction unit 93.

[0229] Based on signals supplied from the preferred range setting unit91 and indicative of a preferred range, data rate information suppliedfrom the data amount computation unit 34 in FIG. 6 and informationsuppled an object detector 95 which will further be described later, theselection controller 92 controls selection by the MUX 32 of each ofbackground image, object image and appended information. That is,receiving the signals indicative of the preferred range from thepreferred range setting unit 91, the selection controller 92 willcontrol the multiplexing in the MUX 32 so that the spatial resolution ofan image inside the preferred range is improved at the sacrifice of thetime resolution of the image. Also, receiving label informationindicative of an object detected by the object detector 95, theselection controller 92 controls the MUX 32 to improve the spatialresolution of the labeled object by the sacrifice of the time resolutionof the image. Further, supplied with data rate of the multiplexed dataoutput from the MUX 32 from the data amount computation unit 34, theselection controller 92 controls the MUX 32 to select data so that thedata rate will not exceed the transmission rate of the transmissionline.

[0230] The feature extraction unit 93 is supplied with background imagedata, object image data and appended information from the pre-processor12, and with signals indicative of a preferred range, from the preferredrange setting unit 91, to extract a feature of an image inside thepreferred range set by the preferred range setting unit 91. That is, thefeature extraction unit 93 will extract, concerning for example anobject inside the preferred range, a feature reflecting a tendency ofthe user of the receiver 2 to be interested in the object.

[0231] More particularly, the feature extraction unit 93 extracts,concerning an object indicating a certain “person” for example, featuresindicating that the object is a “person”, that his or her motion isuniform for example, the object is at this side, that the object is inthe middle position on the screen, that the object is moving at a speed(the object is a moving portion), that for example eyes, nose and mouthare included in areas defining the object (areas of the object includethe eyes, nose and mouth), that the pattern of the object is striped forexample (the object is a striped portion), that the object is red (theobject is a red portion), etc., as shown in FIG. 17.

[0232] Further the feature extraction unit 93 determines a vectordefined by the above extracted features of the object (feature vector),and increments by one the degree of the thus determined feature vectorof a histogram stored in a histogram storage unit 94 also included inthe controller 35.

[0233] The histogram storage unit 94 stores a histogram of a featurevector determined by the feature extraction unit 93 as a result oflearning the preference of the user of the receiver 2.

[0234] The controller 35 further includes an object detector 95. Thisobject detector 95 is provided to detect, from an object image suppliedfrom the pre-processor 12, an object from which there can be determineda similar feature vector to the most frequent feature vector in thehistogram stored in the histogram storage unit 94. That is, the objectdetector 95 determines a feature vector as in the feature extractionunit 93 for an object supplied from the pre-processor 12. Further, theobject detector 95 refers to the histogram stored in the histogramstorage unit 94 to judge whether there exists a feature vector of theobject from the pre-processor 12 within a predetermined range of featurevector space about the most frequency feature vector. When the objectdetector 95 judges that such a feature vector exists, it will predictthat the user of the receiver 2 tends to be interest in the object, andsupply a label indicative of the object to the selection controller 92.

[0235] Next, how the MUX 32 is controlled by the controller 35 which isoperative as having been described with reference to FIG. 16, will bedescribed below with reference to the flow chart in FIG. 18.

[0236] First in step S8 1, the preferred range setting unit 91 judgeswhether click data have been sent from the receiver 2. If it is judgedin step S81 that clock data have been sent from the receiver 2, thepreferred range setting unit 91 goes to step S82 where it will set apreferred range as having previously been described and supply signalsindicative of the preferred range to the selection controller 92 andfeature extraction unit 93.

[0237] In step S83, the selection controller 92 controls the MUX 32 toimprove the spatial resolution of an image (object image) inside thepreferred range at the sacrifice of the time resolution of the image.

[0238] In step S84, the feature extraction unit 93 extracts a feature ofthe object inside the preferred range to determine a feature vector fromthe extracted object feature. Further, the feature extraction unit 93increments by one the degree of the feature vector of the histogramstored in the histogram storage unit 94 in step S85, and then theprocedure returns to step S81.

[0239] The operations in steps S81 to S85 are repeated to form, in thehistogram storage unit 94, histogram of the feature of the object inwhich the user of the receiver 2 tends to be interested. Thereby, thepreference of the user of the receiver 2 will be learned.

[0240] On the other hand, if it is judged in step S81 that no click datahave been sent from the receiver 2, the procedure goes to step S86 wherethe object detector 95 will determine a feature vector of object imagedata supplied from the pre-processor 12 as in the feature extractionunit 94. Further, the object detector 95 refers, in step S87, to thehistogram stored in the histogram-94 to judge whether the feature vectorof the object image supplied from the pre-processor 12 exists inside apredetermined range of the feature vector around the most frequentfeature vector. That is, it is judged in step S87 whether the distancebetween the most frequent feature vector and that of the object imagesupplied from the pre-processor 12 is less than a predetermined value.

[0241] If it is judged in step S87 that the distance between the mostfrequent feature vector and that of the object image supplied from thepre-processor 12 is not less than a predetermined value, that is, whenthe object image supplied from the pre-processor 12 is judged from thepast tendency to have a low probability of interesting the user of thereceiver 2, the procedure goes to step S88 where the selectioncontroller 92 will control the MUX 32 to display ordinary time andspatial resolutions at the receiver 2, and then the procedure returns tostep S81.

[0242] Also, if it is judged in step S87 that the most frequent featurevector and that of the object image supplied from the pre-processor 12is less than a predetermined value, that is, when the object imagesupplied from the pre-processor 12 has a high probability of interestingthe user of the receiver 2, the object detector 95 will output a labelof the object supplied from the pre-processor 12 to the selectioncontroller 92, and then the procedure goes to step S89.

[0243] In step S89, the selection controller 92 controls the MUX 32 toimprove the spatial resolution of the labeled object image from theobject detector 95 at the sacrifice of the time resolution of the image,and then the procedure returns to step S81.

[0244] Therefore, in this case, the labeled object image output from theobject detector 95 is displayed with a high spatial resolution by thesacrifice of the time resolution at the receiver 2. Subsequently, theobject continues to be displayed with the high spatial resolution.

[0245] As a result, when the object in which the user tends to beinterested, the receiver 2 will subsequently continue to be displayedwith the high spatial resolution even with no operation of the clickdata input unit 24 by the user or automatically.

[0246] Note that the histogram of the feature vector as the result oflearning the preference of the user of the receiver 2, stored in thehistogram storage unit 94 can be reset regularly or irregularly orcorrespondingly to a request from the user of the receiver 2.

[0247] Next, there will be described in detail the flow of operations ofcharging made for data service and data transmission/reception, forexample, for data service to the terminals 1 and 2 or datareception/transmission between the terminals 1 and 2, made via themanagement center 103 in FIG. 1, and the construction of a systemintended for such operations.

[0248] Generally a software is constructed from a fusion of variousprocessing methods (programs). For example, a spreadsheet includes aprogram for simple arithmetic operations as well as a program forstatistic operations. Normally, a package medium such as a CD-ROM or thelike having recorded therein such operating methods forming a software(program; will be referred to as “method” hereunder whereverappropriate) is commercially available, and buying such a package mediumto acquire the license for using the software, the user will be able toutilize the software.

[0249] The present invention provides a system in which the user cannotacquire the license for using a software stored in a package medium whenhe or she purchases the package medium but can acquire the license eachtime he uses the software and also acquire the license for a necessaryone of the functions (methods) included in the software. Therefore, theuser should pay an amount charged for the use of a software and also foran amount charge for the use of a necessary one of the softwarefunctions (methods). Thus, the user can use the methods with lessexpenses than in use of the aforementioned package medium which servesall the software functions collectively.

[0250] Also, various methods are possible for a result of an operation.Therefore, selective use of a higher-level one of various methods forattaining similar objects will assure a more desirable result. In otherwords, use of the more advance method will be a larger benefit to theuser.

[0251] According to the present invention, there is provided a chargingsystem in which of a plurality of difference algorithms, a one the useris going to use is selected, and an amount of money is charged accordingthe level of the thus selected algorithm. That is, the charging systemaccording to the present invention charges the user for an amountcorresponding to a merit the user can enjoy.

[0252] Image communications will be described by way of example. Sincein case of the image communications, use of a higher-level compressionmethod (algorithm) will permit to reduce the entire data amount whilemaintaining a high quality of the data, so it is possible using thehigher-level algorithm when the frequency band for the datacommunications is limited to acquire high-quality images, namely,high-definition, distortion- and noise-free images.

[0253] According to the present invention, a method capable ofcompressing an image with a low rate in image communications whilemaintaining the image quality is defined as a relatively low-levelalgorithm, and a method capable of compressing the image at a high rateis defined as a relatively high-level algorithm, and the user using thehigh-level algorithm will be charged for a larger amount than for usingthe low-level algorithm. Of course, the present invention is not limitedto the image compression algorithm in the image communications but itcan be applied to operations using various algorithms such as datacommunication algorithm, sound processing algorithm, etc. and chargingis made for the level of an algorithm used.

[0254]FIG. 19 shows, as an example of the aforementioned chargingsystem, a basic model of charging made between a processor 100 as aterminal having the above-mentioned method pre-installed therein and themanagement center 103. Note that charging for data service to theterminals 1 and 2 and actual data transmission/reception between theterminals 1 and 2, made via the management center 103 shown in FIG. 1,will further be described later.

[0255] As shown in FIG. 19, the processor 100 includes an algorithmprocessor 102 consisting of a plurality of processors 102 ₁ to 102 _(N)which process data with different kinds of algorithms (programs),respectively, and a usage management unit 101 to manage use ofalgorithms by the processors 102 ₁ to 102 _(N). Note that in the exampleshown in FIG. 19, the algorithms of the processors 102 ₁ to 102 _(N) inthe algorithm processor 102 cannot freely be used by the user.

[0256] As also shown in FIG. 19, the management center 103 includes anauthentication memory 104 to hold authentication information on whetherthe processor 100 is an authorized one (the user is an authorized one),a charge computation system 106 to compute a charge for the use of analgorithm by the user based on a history of using the algorithm by theuser (processor 100) and usage-charge computation table 107, and amanagement system 105 to manage the entire system.

[0257] As will be seen from FIG. 19, when the user wants to use any ofprocesses to be effected in the processors 102 ₁ to 102 _(N) of theprocessor 100, he or she will first make a selective input to select oneof the processors 102 ₁ to 102 _(N) for use (that is, one of algorithmsto be used). When the user make the input, the usage management unit 101will first make an inquiry to the management center 103 for permissionto use the process (algorithm).

[0258] Receiving the permission inquiry signal, a management system 105provided in the management center 103 checks, based on authenticationinformation stored in the authentication memory 104, whether theprocessor 100 having made the inquiry is a one under the managementsystem 100, that is, whether the processor 100 having made the inquiryis an authorized one (authorized user). When the processor 100 ischecked to be an authorized processor (authorized user), the managementsystem 105 sends an algorithm-use permission signal to the processor100.

[0259] Upon reception of the algorithm-use permission signal, the usagemanagement unit 101 in the processor 100 controls the algorithmprocessor 102 to immediately start a process in a processorcorresponding to a process for which a selection input has been made bythe user (a process of which the use is allowed by the management center103). Thus, a user's desired operation will be effected.

[0260] Also, the management system 105 in the management center 103sends, to the charge computation system 106, information (usage history)indicative of a processor 100 (user) to which the algorithm-usepermission has been sent and a process (algorithm) of which the use hasbeen permitted. Receiving the usage history information, the chargecomputation system 106 uses a usage-charge computation table held in thecharge computation table 107 and set for each algorithm to compute acharge for the use of the process (algorithm). Thereafter, the chargecomputation system 106 will send a request for payment of the chargedamount to the user of the processor on an on-line or off-line basis.

[0261]FIG. 19 shows an example in which the method (program) ispre-installed in the processor 100. As in a basic model for chargingshown in FIG. 20, however, a processor 10 holds no method (program) buta method may be acquired from a server such as the management center 103or the like (program is down-loaded at each time a process has beeneffected).

[0262] As shown in FIG. 20, the processor 110 has provided therein amemory 111 to provisionally hold a method (program) sent from themanagement center 103, and an MPU (microprocessor unit) to make aprocess by a method (program) held in the memory 110.

[0263] The management center 103 shown in FIG. 20 includes anauthentication memory 104 to hold authentication information on whetherthe processor 110 is an authorized one (the user is an authorized one),a charge computation system 106 to use a history of method (program)sending to the user (processor 110) and a usage-charge computation tableheld in a charge computation table 107 for each algorithm, an algorithmmemory 115 to hold a plurality of difference algorithms (program), and aprocess management processor 114 to manage the entire system, read andcompress, for example, a program from the algorithm memory 115 uponrequest from the processor 110, and send the program to the processor110.

[0264] As shown in FIG. 20, to use any of operations corresponding toalgorithms, respectively, held in th algorithm memory 115 of themanagement center 103, the user of the processor 110 will first give adesignation for selection of any of the processes. Given the user'sdesignation, the MPU 112 of the processor 10 makes a request to themanagement center 103 for a processing method (algorithm).

[0265] Upon reception of the request for sending of the algorithm, theprocess management processor 114 of the management center 103 checks,based on the authentication information stored in the authenticationmemory 104, whether the processor 110 having made the request is aprocessor under the management of the process management processor 114,that is, whether the processor 110 having made the request is anauthorized one (authorized user). Making sure in the authentication thatthe processor 110 is an authorized one (authorized user), the processmanagement processor 114 reads an operating procedure (program) from thealgorithm memory 115, compresses, for example, the program and encryptsit as necessary, and sends it to the processor 110.

[0266] Receiving the compressed and encrypted program sent from theprocess management processor 114, the MPU 112 of the processor 110defreezes and decrypts the program and stores the program into thememory 111. Thereafter, the MPU 112 uses the program stored in thememory 111 to make an operation. Thus, the user's desired operation willbe effected. Note that after thus effected, the program stored in thememory 111 will destruct itself or automatically be erased againstre-use.

[0267] Also, the process management processor 114 of the managementcenter 103 sends, to the charge computation system 106, information(usage history) indicative of the processor (user) having sent theprogram and method for effecting the program (algorithm). Receiving theusage history, the charge computation system 106 uses th table held inthe charge computation table 107 to compute a charge for the use of theprocess (algorithm). Thereafter, the charge computation system 106 willsend a request to the user of the processor 110 for payment of thecharge on an on-line or off-line basis.

[0268] Next, there will be described with reference to each of FIGS. 21and subsequent drawings a charging which will be made in case theaforementioned charging model is applied to an actual charging system,that is, for example a charging for data service to the terminals 1 and2 and also for data transmission/reception between the terminals 1 and2, made via the management center 103 in FIG. 1.

[0269] In the example shown in FIG. 1, the data mount of an image sentand received is a problem as having previously been described, and inthe aforementioned example, the data amount is controlled by dividing animage into a background, object and appended information andhierarchically encoding them. In the following example, however, forcontrolling the data amount, an appropriate one corresponding to auser-demanded image quality is selected from among different imagecompression algorithms and image data compressed by the use of theselected algorithm is sent and received. Note that in this embodiment,the different algorithms for the image compression include algorithmsfor sub sample compression, object encoding-compression,classification-adaptive predictive compression and others (MPEGcompression, for example) in addition to the hierarchical encodingalgorithm.

[0270] According to the present invention, the above algorithms furtherinclude, in addition to the above image compression algorithms,algorithms in all layers (software layer) such as an algorithm forautomatic selection (optimization) of any of the image compressionalgorithms correspondingly to a situation, an algorithm for predictingan area (focus of interest) the user is interested in in an image bylearning as having previously been described with reference to FIGS. 16to 18, an algorithm for automatic control of the power consumption ofthe system. That is, the software used in the system consists of forexample a hardware control software, OS (operation system), applicationsoftware, etc. In this embodiment, however, it is possible to selectdifferent algorithms in the application software layer such as the imagecompression algorithms, automatic algorithm selection (optimization)algorithm and interesting-focus predictive algorithm as well asdifferent algorithms in the hardware control software layer such as theautomatic power-consumption control algorithm and a control algorithmfor control of switching between various kinds of hardware.

[0271] According to the present invention, a desired one is selectedfrom among the plurality of different algorithms, a charging will bemade for the selected algorithm. Note that the amount charged for eachalgorithm is defined by a sum of charges for algorithm selections madein respective layers.

[0272]FIG. 21 shows, by way of example, the construction of theessential portion of a system in which an appropriate algorithm selectedfrom among the different image compression algorithms correspondingly tothe image quality demanded by the user can be used to compress imagedata and send/receive the compressed image data. For the simplicity ofthe illustration and explanation, FIG. 21 shows only the essentialportion of the system configured by applying the charging modelaccording to this embodiment to the system shown in FIG. 1, withomission of the other construction of the transmitter 1 and receiver 2in FIG. 1. The transmitter 1 and receiver 2 are basically identical inconstruction to each other. However, FIG. 21 shows the constriction ofonly the essential portion of the transmitter 1 for sending image datain response to a sending request from the receiver 2 and also theconstruction of only the essential portion of the receiver 2 forreceiving a user's request and image data sent in response to therequest.

[0273] As shown in FIG. 21, the transmitter 1 includes mainly the videocamera 6 (equivalent to the aforementioned image input unit 11) foracquisition of image data, an image compression processor 121 havingprovided therein a sub sample compression unit 122 to effect the subsample compression algorithm, object encoding-compression unit 123 toeffect object ecoding-compression algorithm, classification-adaptivepredictive compression unit 124 to effect the classification-adaptivepredictive compression and the hierarchical encoder 125, an image datatransmission unit 127 to send compressed image data, a request receivingunit 126 for receiving a signal of request for sending image data, and acontroller 128 to control, upon reception of the request for image datasending, the image compression processor 121 in response to the sendingrequest, automatically select (optimize) an optimum one of the imagecompression algorithms as necessary and predict a focus of user'sinterest.

[0274] Also as shown in FIG. 21, the receiver 2 includes a user'sdesignation input unit 132 to input a selection by the user (input forimage quality selection in this embodiment), a controller 133 to selectany of the image compression algorithms based on a selection entered bythe user and generate a signal for requesting any one of the algorithmfor automatic selection (optimization) of an image compression algorithmaccording to a situation, algorithm for prediction of an area (focus ofinterest) the user is interested in in an image, algorithm for automaticcontrol of the power consumption of the system, etc., a requestsending/algorithm-use permission reception unit 134 to send the requestsignal to the management center 103 and receive an algorithm-usepermission signal from the management center 103, a power consumptioncontroller 131 to control the system power consumption when theautomatic power-consumption control algorithm, an image data receptionunit 135 to receive the compressed image data sent from the managementcenter 103, a compressed data decoder 136 to decode the receivedcompressed image data correspondingly to the algorithm used to compressthe received image data, and a display unit 137 (equivalent to thedisplay unit 7) to display the decoded image.

[0275] Note that the system shown in FIG. 21 has pre-stored therein theplurality of methods as in the model shown in FIG. 19 but it may beadapted to acquire a method from the management center 103 whennecessary as in the model shown in FIG. 20. Therefore, the managementcenter 103 will have any or both of the constructions shown in FIGS. 19and 20 depending upon which is adopted, the model in FIG. 19 or 20.

[0276] As shown in FIG. 21, when the receiver 2 acquires image datapicked up by the video camera 6 of the transmitter 1, a user'sdesignation signal is supplied from the user's designation input unit132 of the receiver 2. Note that the criteria of the designations forinput of a user's designation (criteria for selection of an algorithm)include a demanded image quality (performance of processing result),processing speed, processing time, processing result, charge for thealgorithm use, power consumption, etc.

[0277] Also, the function of designating a position in an image by mouseclicking, disclosed in the aforementioned PCT Published UnexaminedApplication No. WO01/11889A1 is owned by the user's designation inputunit 132. This function can also be used to predict a focus of user'sinterest and designate only an image block around a point designated bymouse clicking as a data area to be subjected to signal processing.

[0278] Supplied with a designation signal for requesting a desired imagequality from the user's designation input unit 132, the controller 133generates a signal for requesting an algorithm for an image compressionpermitting the image quality corresponding to the designation signal andsends it to the request sending/algorithm-use permission reception unit134. The request sending/algorithm-use permission reception unit 134sends the request signal to the management center 103 via a basestation, exchange or the like (not shown). Note that it is assumed thatan initialization such as securing a communication path has been endedat this time.

[0279] Upon reception of the request signal, the management center 103checks whether the receiver 2 having send the request signal is underthe management of the management center 103, that is, whether thereceiver 2 having sent the request signal is an authorized one(authorized user). When the receiver 2 is checked to be an authorizedone, the management center 103 sends the request signal to thetransmitter 1 via the base station, exchange or the like (not shown) andan algorithm-use permission signal to the receiver 2.

[0280] Receiving the selection signal, the request reception unit 126 ofthe transmitter 1 sends the request signal to the controller 128. Thecontroller 128 will analyze the content of the selection signal andselect, based on the result of the analysis, any of the sub samplecompression, object encoding-compression, classification-adaptivepredictive compression and hierarchical encoding algorithms forcompression of image data acquired by the video camera 6, and put anyone of the units 122 to 125, corresponding to the selected algorithm.Thus, the image data compressed by the image compression processor 121is sent to the image data transmission unit 127. The image datatransmission unit 127 sends the compressed image data to the receiver 2via the base station, exchange or the like (not shown) under the controlof the controller 128. Note that the compressed image data supplied fromthe transmitter 1 may be sent to the management center 103 once, andthen it may be sent from the management center 103 to the receiver 2 viathe base station, exchange or the like (not shown).

[0281] Receiving the compressed image data, the image data receptionunit 135 of the receiver 2 sends the received compressed image data tothe compressed image data decoder 136. Under the control of thecontroller 133, this compressed image data decoder 136 uses a decodingalgorithm corresponding to the image compression algorithm selected andallowed in advance for use to decode the compressed image data, andsends the decoded image data to the display unit 137. Thus, the displayunit 137 will display thereon an image picked up by the video camera 6of the transmitter 1.

[0282] In the receiver 2, when supplied, from the user's designationinput unit 132, with a user's designation signal for designating thealgorithm for automatic selection (optimization) of any of the imagecompression algorithms according to a situation or the algorithm forprediction of an area (focus of interest) the user is interested in inan area, the controller 133 will generate, based on the designationsignal from the user's designation input unit 132, a signal of requestfor the automatic algorithm selection (optimization) algorithm orinteresting-focus prediction algorithm, and send it to the requestsending/algorithm-use permission reception unit 134. Thus, the requestsending/algorithm-use permission reception unit 134 will send therequest signal to the management center 103.

[0283] The management center 103 authenticates the request signal, andthen sends the request signal to the transmitter 1 and algorithm-usepermission signal to the receiver 2.

[0284] The controller 128 in the transmitter 1 analyzes the content ofthe request signal, automatically selects any of the sub samplecompression, object encoding-compression, classification-adaptivepredictive compression and hierarchical encoding algorithms and predictsa focus of user's interest (user-interesting object image), based on theresult of analysis, to control the image compression processor 121 toimprove the image quality by making the compression rate higher for theuser-interesting object image than for the other images. Thus, the imagedata compressed by the image compression processor 121 is sent from theimage data transmission unit 127 to the image data reception unit 135 ofthe receiver 2 via the base station, exchange or the like.

[0285] At this time, the compressed image data decoder 136 of thereceiver 2 will use the decoding algorithm corresponding to the imagecompression algorithm automatically selected by the automatic algorithmselection algorithm, under the control of the controller 133, to decodethe compressed image data and send the image data to the display unit137.

[0286] At the receiver 2, when supplied with a user's designation signalfor designating the automatic power-consumption control algorithm fromthe user's designation input unit 132, the controller 133 sends aninquiry signal for permission of using the automatic system powerconsumption control algorithm to the management center 103 via therequest transmission unit 134. Receiving a permission signal in responseto the inquiry signal from the management center 103, the receiver 2will be able to use the power consumption control algorithm and thepower consumption controller 131 will automatically control the powerconsumption of this system.

[0287] Simultaneously with or after the aforementioned operation, themanagement center 103 uses the receiver (user) having sent thealgorithm-use permission and usage history information on processes ofwhich the use has been permitted, namely, on the image compressionalgorithms, automatic algorithm selection (optimization) algorithm,interesting-point prediction algorithm, automatic power-consumptioncontrol algorithm, etc. to compute a charge to the user of the receiver2 for the use of the algorithms and request the user for payment of thecharged amount. Thus, the user of the receiver 2 will pay, to themanagement center 103, an amount corresponding to the request forpayment of usage charge Note that in the example in FIG. 21, theoperations made according to each image compression algorithm, automaticalgorithm selection (optimization) algorithm and to user-interestingpoint prediction algorithm are actually effected in the transmitter Ibut since it is the receiver 2 that selects such algorithms and benefitsby them, the charge for the use of the algorithms will be made to thereceiver 2 (user) benefits by the algorithms.

[0288] The charge for the use of the algorithms is defined by a sum ofcharges made for requests in the respective layers. Therefore, in caseonly the image compression algorithms have been used, there will becomputed a charge preset for the use of each of the sub samplecompression, object encoding-compression, classification-adaptivepredictive compression and hierarchical encoding algorithms. Also, incase for example the automatic algorithm selection (optimization)algorithm has been used, charging will be made with a charge for the useof an image compression algorithm selected according to the automaticalgorithm selection (optimization) algorithm, added to the charge forthe use of the automatic algorithm selection (optimization) algorithm.Similarly, when the user-interesting point prediction algorithm andautomatic power-consumption computation algorithm have been used,charging will be made with addition of the charges for the use of thesealgorithms. Note that a request for payment of the charges may be madeon the basis of the result of charge computation to the userconsecutively or collectively for a certain period. Also the chargingmay be made proportionally to a time for which the algorithms have beenused.

[0289] In this embodiment, a request for payment of a charge is made tothe user of the receiver 2 from the management center 103 on an on- oroff-line basis, and the user responds to the request to pay the chargedamount to a designated account or the like on an on- or off-line basis.Alternatively, the management center 103 may be adapted to havepre-stored therein a number of account for each identifier of the userof the receiver 2 and a number of account of thereof, and the chargecomputation system 106 may be adapted to compute a charge to the usercorrespondingly to the request signal from the user of the receiver 2and send, to an external payment sensor, at least the user's number ofaccount, account number of the management center 103 and money amountinformation corresponding to an amount charged to the user, whereby thecharge can instantly be paid.

[0290] Note that the user of a receiver has only to designate hisrequest such as a request for the image quality for example, and may notalways be conscious of his own selection of algorithms.

[0291] Next, FIG. 22 shows a flow of operations of the transmitter 1 forexample in FIG. 21.

[0292] As shown in FIG. 22, first in step S 101, the request receptionunit 126 waits for reception of a signal of request for the algorithms,and upon reception of the request signal, it will send the requestsignal to the controller 128.

[0293] Receiving the signal of request for the algorithms, thecontroller 128 controls, in step S102, the image compression processor121 to make a selection between algorithms. That is, the controller 128controls the image compression processor 121 to use any one of the units122 to 125, which corresponds to an algorithm designated by the requestsignal.

[0294] When an algorithm is selected by the controller 128, the imagecompression processor 121 encodes image data (by compression) by theunit for the designated and selected algorithm, and sends theencoding-compressed image data obtained to the image data transmissionunit 127, in step S103.

[0295] In step S104, the image data transmission unit 127 sends thecompressed image data from the image compression processor 121 to thereceiver 2 via the base stations 3 and 5, exchange 4 and managementcenter 103.

[0296] Thereafter, the transmitter 1 (controller 128 for example)judges, in step S 105, whether the process is to be ended. If thejudgement is that the process is not to be ended, the transmitter 1 goesback to step S101. If it is judged that the procedure is to be ended,the process in the transmitter 1 in FIG. 22 is ended.

[0297] Next, FIG. 23 shows a flow of operations of the receiver 2 inFIG. 21.

[0298] As shown in FIG. 23, first when a user's designation signal fordesignation of for example the image quality is supplied from the user'sdesignation input unit 132 and a request signal (signal indicative of analgorithm to be used) is generated by the controller 133, the requestsignal is sent, in step S110, from the request sending/algorithm-usepermission reception unit 134 to the management center 103.

[0299] After sending the request signal, the controller 133 checkswhether the request sending/algorithm-use permission reception unit 134has received the algorithm-use permission signal from the managementcenter 103. When the request sending/algorithm-use permission receptionunit 134 has received an algorithm-use inhibition signal or has notreceived the algorithm-use permission signal in a predetermined time,the controller 133 exits the procedure. On the other hand, when therequest sending/algorithm-use permission reception unit 134 has receivedthe algorithm-use permission signal from the management center 103, itwill be supplied with compressed image data from the transmitter 1.

[0300] The image data reception unit 135 waits for arrival of compressedimage data. Upon reception of compressed image data in step S112, theimage data reception unit 135 will send the compressed image data to thecompressed image data decoder 136.

[0301] Upon reception of the compressed image data, the compressed imagedata decoder 136 decodes the compressed image data by reverselyfollowing the image data compressing procedure, and sends the decodedimage signals to the display unit 137. Thus, an image is displayed onthe display unit 137. The above are done in step S1 13.

[0302] Thereafter in step S114, the controller 133 judges whether arequest for exiting the procedure has been sent from the user'sdesignation input unit 132. If no ending request has not been made sentfrom the user's designation input unit 132, the procedure returns tostep S110. If the ending request has been sent, an ending signal isgenerated in step S115 and sent to the management center 103 via therequest sending/algorithm-use permission reception unit 134. Themanagement center 103 will have the transmitter 1 end the transmission,whereby the procedure is ended.

[0303]FIG. 24 shows a flow of operations of the management center 103shown in FIGS. 21 and 19. Note that the operations will be explainedwith reference to the internal construction of the management center 103in FIG. 19 wherever appropriate.

[0304] First in step S121 in FIG. 24, the management system 105 waitsfor arrival of a signal of request for an algorithm (algorithmrequesting signal) from the receiver 2. Upon reception of the requestsignal, the management system 105 uses, in step S122, information heldin the authentication memory 104 to judge whether the request from thereceiver 2 is valid or not. If the request from the receiver 2 is judgedin step S122 to be invalid, the management system 105 sends an inhibitsignal as a refusal signal, and then exits the procedure.

[0305] On the other hand, if in step S122 the request is judged to bevalid, the management system 105 sends the request signal (signalindicative of a requested algorithm) to the transmitter 1, andinformation indicative of the requested algorithm and receiver 2 to thecharge computation system 106.

[0306] Receiving the information indicative of the request algorithm andreceiver 2 from the management system 105, the charge computation system106 uses, in step S125, the information and the usage-charge computationtable in the charge computation table 107 to compute a charge.

[0307] Thereafter, the management system 105 judges whether an endingsignal has arrived from the receiver 2. When no ending signal has beenreceived, the procedure returns to step S121 to continue the operation.

[0308] In step S126, it is judged whether the transmission is to beended or continued. If the judgment in step S127 is “end”, the chargecomputation system 106 sums the amounts charged for the use ofrespective processes, and requests, in step S 128, the user of thereceiver 2 for payment of the sum of charged amounts. The request maynot be made on an on- or off-line basis to the user but may be send theuser's number of account, account number of the management center 103and charged amount information to an external payment center under thecontrol of the controller 133.

[0309] Thereafter, the management system 105 sends an ending signal tothe transmitter 1 to end the process in FIG. 24. Note that operations insteps S127 and S128 may be done after completion of an operation in stepS129.

[0310] Next, a flow of operations made according to the automaticalgorithm selection (optimization) algorithm will be described withreference to the flow chart in FIG. 25.

[0311] The algorithm selection or optimization method varies dependingupon on what the selection or optimization is effected. In the abovedescription, the user selects an image compression algorithm on thebasis of image quality which can be attained by the algorithm. However,there will be described an example of process (algorithm) selection madeon the basis of how low the usage charge is for the use of the automaticalgorithm selection (optimization). That is, according to the automaticalgorithm selection (optimization) algorithm, there will first beselected an algorithm whose usage charge is lowest. Note that althoughin this embodiment, an image compression algorithm is automaticallyselected but the present invention is not limited to this example butany algorithm may automatically be selected (optimized) while thecriterion of optimization is not limited to the usage charge but it maybe image quality, processing time, data transfer time or the like.

[0312] When a designation to select an algorithm for optimization ofimage compression for example is supplied from the receiver 2, thecontroller 128 in the transmitter 1 in FIG. 21 will select, in step S171in FIG. 25, a one, which can be used with the lowest charge, of the subsample compression, object encoding-compression, classification-adaptivepredictive compression and hierarchical encoding algorithms. Thus, inthe image compression processor 121, image data are compressed by acompression unit corresponding to the algorithm of which the usagecharge is smallest. It is assumed here that the usage charge isdetermined on the basis of the level, for example, of the image datacompression and that the usage charge for the sub sample compressionalgorithm which provides the simplest compression is 100 yens per use,that for the object encoding-compression algorithm which provides a nextsimplest compression is 200 yens per use, that for the hierarchicalencoding algorithm which provides a higher-level process is 300 yens peruse and that for the classification-adaptive predictive compressionalgorithm which provides the highest-level process is 400 yens per use.In this case, since the usage charge for the sub sample compressionalgorithm is smallest, the sub sample compression unit 122 in the imagecompression processor 121 is selected in step S171 to compress the imagedata.

[0313] Next, in step S172, in the image compression processor 121computes a difference between an original image and an image obtained bydecoding (by a local decoder) data having been compressed according tothe selected image compression algorithm, and returns the difference tothe controller 128.

[0314] Supplied with the difference, the controller 128 will judge, instep S 174, whether the difference is smaller than a predeterminedthreshold.

[0315] If the difference is not smaller than the predeterminedthreshold, it means that the quality of an image resulted from thecompression with the selected image compression algorithm is lower thanthe limit. Therefore, in this case, the controller 128 will select animage compression algorithm which will assure better image quality. Ifthe sum sample compression algorithm has been selected during theprecedent selection of image data compression algorithm, the controller128 will select the object encoding-compression algorithm whose level ishigher than the sub sample compression algorithm. Thereafter, in theimage compression processor 121, the newly selected algorithm is used tocompress image data, and then a difference between the original imageand image resulted from the compression, and the controller 128 willjudge again whether the difference is lower than the predeterminedthreshold. This series of operations will be repeated until thedifference becomes smaller than the predetermined threshold.

[0316] If the difference is judged in step S174 to be smaller than thepredetermined threshold, the image compression processor 121 will send,in step S175, image data having been compressed according to theselected image compression algorithm via the image data transmissionunit 127.

[0317] Assume here that the compression parameters in each of the imagecompression algorithms take default values. That is, with thecompression parameters (quantization coefficient, quantization bit,sampling frequency and bit rate) being as default values, thecompression ratio of a natural image in each algorithm will be set to ½or so.

[0318] Thereafter, the controller 128 will judge, in step S176, whetheran ending request has been made from the receiver 2. If the request hasnot arrived, the procedure returns to step S171. If the request has beenmade, the process is ended. The above are the operations which will bemade when it is designated that the automatic algorithm selection(optimization) algorithm is should be selected.

[0319] In this case, the receiver 2 will be requested to pay a sum of anamount charged for the use of the image compression algorithm havingbeen selected when the difference is smaller than the predeterminedthreshold and an amount charged for the use of the automatic algorithmselection (optimization) algorithm.

[0320] Note that the predetermined threshold on which the image qualitydepends can be determined in a plurality of manners. For example, athreshold may be pre-determined or a processed or compressed image isprovided to the user for determination of a threshold.

[0321] As having been described in the above with FIG. 25, an imagecompression algorithm finally selected through study of the imagequality obtained by compressing an image once (by comparison inperformance) is taken as an optimum algorithm for compression of imagedata. By detecting feature of an image in advance, however, it is may bejudged based on the image feature which one is most optimum. It is notexpectable that use of a simple compression algorithm with for examplean image whose values of brightness, motion, color, etc. vary greatly(high-activity image) will assure sufficient quality of the image. Tosolve this problem, a higher-level compression algorithm mayautomatically be selected for such an image. Also, a suitablecompression algorithm for an image can automatically be selected bydetecting as a feature of the image whether the image is a CG (computergraphics) image or whether the image is a computer-game image. In thiscase, since image compression is not tried repeatedly, so the processingspeed can be higher than in the method in which the image compressionalgorithms are selected one after another starting with the one whoseusage charge is the lowest. By presetting a relation between an imagefeature and compression algorithm, it is possible to speedily select anoptimum compression algorithm.

[0322] Next, of the aforementioned image compression algorithms, the subsample compression algorithm will further be described. The sub samplecompression algorithm is such that the number of pixels forming an imageis reduced by thinning out them at a rate of one per two pixels forexample. Of course, the thinning-out of pixels may be done at a rate ofone per three, one per four or the like.

[0323] Next, the object encoding included in the image compressionalgorithms will be described.

[0324] The object encoding is such that a portion of an image which doesnot vary so much as the time elapses is extracted as an object image andinformation on the object image is sent once but not sent thereafterwhile position information (and motion information) on the object imageis sent, thereby compressing the information. Note that the object maybe regarded as generally the same as the object having previously beendescribed with reference to FIG. 2.

[0325] In sending for example a moving picture, it will lead to a wasteof a transmission band to send image data on an object image which willnot vary at all. Therefore, by not sending again an object image datasent once, the information can be sent in a considerably reduced amount.

[0326] The object encoding will further be described with reference toFIG. 26. For this explanation, it is assumed that there is a movingimage consisting of 30 frames/sec, each frame including a background 151consisting of a stationary sky and mountain, a moving cloud 152 and amoving car 153 as shown in FIG. 26A. As shown in FIG. 26B, an objectimage of the background 151, an object image of the car 153 and anobject image of the cloud 152 will be extracted from one frame of theimage. By this object encoding, a movement vector indicative of adirection and amount (distance) of an image movement can be determinedfor each object by block matching between image frames in FIG. 26A. Inthe example shown in FIG. 26B, it is assumed that the movement vector ofthe stationary background 151 is (x, y)=(0, 0), the movement vector ofthe car 153 is (x, y)=(−2, 0) indicating that the car 153 moves by −2 inthe x-direction while it does not move in the y-direction and themovement vector of the cloud 152 is (x, y)=(3, 2) which indicates thatthe cloud 152 moves by 3 and 2 in the x- and y-directions, respectively.With the object encoding, information on the shape of each of theobjects including the background 151, car 153 and cloud 152 is acquiredand sent, and then the initial position information on each object andmovement vector of each frame is sent or the position information oneach frame is sent, whereby the amount of data to be sent canconsiderably be reduced. On the other hand, at the receiving side, animage is reproduced from the shape information on the objects andposition information on each frame or movement vectors.

[0327]FIG. 27 shows a flow of operations of the objectencoding-compression unit 123 in FIG. 21 for the above object encoding.

[0328] As will be seen in FIG. 27, first, when the compression by theobject encoding-compression unit 123 is selected by the controller 128,the object encoding-compression unit 123 makes an initial processing instep S131. By the initial processing, initial image data are acquiredfrom the video camera 6, objects included in the image are extracted asobject images, and data (shape information etc.) for use in subsequentprocesses are taken into an object table and held.

[0329] After the initial processing in step S131, the objectencoding-compression unit 123 goes to step S132 where it will acquireto-be-processed image data one after another from the video camera 6,and further goes to step S133 where it will acquire information onobject images already held in the object table, such as unit-time (ex.one frame) earlier images.

[0330] Next, in step S134, the object encoding-compression unit 123divides the current image into object images corresponding to objectimages held in the object table. In this process, for example portionsidentical in color to the object images held in the object table aretaken as the same images, and objects are extracted based on thepreviously described click data.

[0331] Then, in step S135, the object encoding-compression unit 123extracts, by computation, information on movement (movement vector)between object images held in the object table and those in the currentframe, and sends the movement vector along with the initial positioninformation to the receiver 2 in step S136. In the aforementionedexample, the movement vector is determined by the block matching. Inthis process, however, since the object images are already extracted andheld in the object table in step S134, it is possible to accuratelydetermine the movement by matching each of the object images with theobject images held in the object table.

[0332] In step S137, the object encoding-compression unit 123 checks ifthere have appeared new object images other than the object imagesalready extracted and whose information are held in the object table. Ifno new object images are detected in this step S137, the objectencoding-compression unit 123 goes to step S140. On the contrary, if anynew object images are detected, the object encoding-compression unit 123goes to step S138 where it will add information on the new object imagesto the object table, and further goes to step S139 where it will send,to the receiver 2, the information on the new object images (such asobject image data, initial position information, movement vectorinformation, etc.). After the process in this step S139, the objectencoding-compression unit 123 goes to step S140.

[0333] In step S140, the object encoding-compression unit 123 stores,for a next process, object images in the current frame and positioninformation or movement vector information on the object images to theobject table. Then, it goes to step S141 where it will judge whether anending request has been sent from the receiver 2. When the endingrequest has not been sent, the procedure returns to step S132. When theending request has been sent, the object encoding-compression unit 123exits the process. The flow of operations for compression of an objectby encoding has been described in the above.

[0334] Next, the construction and operations of theclassification-adaptive predictive compression unit 124 in FIG. 21 willfurther be described.

[0335] First, description of an image with the use of a time-space modelwill be described herebelow with reference to FIG. 28.

[0336] In FIG. 28, T1 and T2 indicate two frames temporally continuouswith each other. T1 indicates a past frame while T2 indicates a currentframe. In these frames, there are shown pixels used to generate andclassify an estimated value. In FIG. 28, a future pixel indicated with asymbol “x” in the current frame is represented by a linear primarycombination model of the number n of taps of values xi and coefficientswi (where i=1, 2, . . . , n) shown in FIG. 28. The coefficient is apredictive coefficient. A set of coefficients is defined for each frameor class so that a square of a difference from a true value of anestimated value y of a future pixel represented by the linear primarycombination of the values xi ad coefficients wi of the pixels is minimumas the result of a computation by the least-square method.

[0337] In the time-space model shown in FIG. 28, the number of taps isn=16. The predicted value y of the future pixel in the current frame T2is represented by a linear primary combination model w1x1+w2×2+ . . .+w16×16 of 16 taps of input pixels. For a coefficient wi in the linearprimary combination model, a value is determined which results in aminimum difference between an actual value and estimated valuerepresented by the linear primary combination model.

[0338] For determination of the above yet-to-set coefficient wi, anexpression of the linear primary combination model is generated in whichvalues xi (i=1, 2, . . . , n) of ${X = \begin{bmatrix}{{X_{11}X_{12}},{\ldots \quad X_{1n}}} \\{{X_{21}X_{22}},{\ldots \quad X_{2n}}} \\\cdots \\{{X_{m1}X_{m2}},{\ldots \quad X_{mn}}}\end{bmatrix}},{W = \begin{bmatrix}W_{1} \\W_{2} \\\cdots \\W_{n}\end{bmatrix}},{Y = \begin{bmatrix}y_{1} \\y_{2} \\\cdots \\y_{m}\end{bmatrix}}$

[0339] each pixel shown in FIG. 28, which will be when the input imagesare shifted each by one pixel in the spatial direction (horizontally andvertically) and actual values yj (j=1, 2, . . . , m) of images to bepredicted are placed. For determination of a set of coefficients for oneframe for example, input images are shifted each by one pixel togenerate a great number of expressions, that is, simultaneous equations(will be referred to as “observation equation) for the number m ofpixels in one frame. For 16 coefficients, it is necessary to provide atleast 16 simultaneous equations. The number m of the simultaneousequations can appropriately be selected based on the balance between theaccuracy and processing time. The observation equation is given by thefollowing expression (1) below:

XW=Y  (1)

[0340] where each of X, W and Y is a matrix as given by the followingequations (2):

. . .   (2)

[0341] A coefficient w which will provide a minimum error with respectto an actual value is determined by the least square method. To thisend, a following residual equation (3) is prepared by adding a residualmatrix E to the right side of the observation equation. That is, theleast square method is used to determine a square of elements in theresidual matrix E in the residual equation, namely, a coefficient matrixW which provides a minimum error.

. . .   (3)

[0342] The requirement for finding the most probable value of eachelement wi in the coefficient matrix W from the residual equation (3) isto minimize the sum of squares ${{YW} = {Y + E}},{E = \begin{bmatrix}e_{1} \\e_{2} \\\cdots \\e_{m}\end{bmatrix}}$

[0343] of m residuals corresponding to the pixels in an image block. Therequirement is given by the following equation (4):

e ₁ ×∂e ₁ /∂w _(i) +e ₂ ×∂e ₂ /∂w ₁ + . . . +e _(m) ×∂e _(m) /∂w _(i)=0(i=1, 2, . . . , n)  (4)

[0344] Then, it suffices to undetermined coefficients w1, w2, . . . ,wn, the elements of the coefficient matrix W, which satisfy the aboverequirements to the number of n. Therefore, the following equations (5)can be obtained from the above residual equation (3):

∂e _(i) /∂w ₁ =x _(i1) , ∂e _(i) /∂w ₂ =x _(i2) , . . . , ∂e _(i) /∂w_(n) =x _(in) (i=1, 2, . . . , n)  (5)

[0345] By setting i=1, 2, . . . , n for the conditions of the equation(4), the following equations (6) are provided: $\begin{matrix}{{{\sum\limits_{i = 1}^{n}e_{ixi1}} = 0},{{\sum\limits_{i = 1}^{n}e_{ixi2}} = 0},\ldots \quad,{{\sum\limits_{i = 1}^{n}e_{{ix}\quad {in}}} = 0}} & (6)\end{matrix}$

[0346] The following normal equation (7) can be obtained from theequations (3) and (6): $\begin{matrix}\{ \begin{matrix}{{{( {\sum\limits_{j = 1}^{m}{X_{j1}X_{j1}}} )W_{1}} + {( {\sum\limits_{j = 1}^{m}{X_{j1}X_{j2}}} )W_{2}} + \ldots + {( {\sum\limits_{j = 1}^{m}{X_{j1}X_{j\quad n}}} )W_{n}}} = ( {\sum\limits_{j = 1}^{m}{X_{j1}Y_{j}}} )} \\{{{( {\sum\limits_{j = 1}^{m}{X_{j2}X_{j1}}} )W_{1}} + {( {\sum\limits_{j = 1}^{m}{X_{j2}X_{j2}}} )W_{2}} + \ldots + {( {\sum\limits_{j = 1}^{m}{X_{j2}X_{j\quad n}}} )W_{n}}} = ( {\sum\limits_{j = 1}^{m}{X_{ju2}Y_{j}}} )} \\\vdots \\{{{( {\sum\limits_{j = 1}^{m}{X_{jn}X_{j1}}} )W_{1}} + {( {\sum\limits_{j = 1}^{m}{X_{jn}X_{j2}}} )W_{2}} + \ldots + {( {\sum\limits_{j = 1}^{m}{X_{jn}X_{j\quad n}}} )W_{n}}} = ( {\sum\limits_{j = 1}^{m}{X_{jn}Y_{j}}} )}\end{matrix}  & (7)\end{matrix}$

[0347] The normal equation (7) is a simultaneous equation includingunknown quantities just to the number of n. Thus, each unknowncoefficient wi being the most probable value can be determined. Moreaccurately, when the matrix as to each undetermined coefficient wi inthe normal equation (7) is normal, the coefficient wi can be determined.Actually, the Gauss-Jordan elimination is used to determine theundetermined coefficient wi. In this way, one set of coefficients forrepresenting future pixels is defined per frame. When mainly frames ofan input image are used to make a learning similar to the aforementionedone, a mean coefficient defined for each frame or a coefficient ofmaximum frequency is stored in a coefficient memory 205 which willfurther be described later.

[0348] In this embodiment, a coefficient determined by the learning isused as a predictive coefficient to make a predictive encoding. In thiscase, the input image is classified (clustered) for an improved accuracyof estimation. In FIG. 28, pixels indicated with a small circle andsmall double circle respectively are ones used in the linear primaryconnection model, and the pixels indicated with the small double circleare ones used in the classification. That is, four of the 16 pixels areused in the classification as well.

[0349] If each of the four pixels is of 8 bits, values of the pixels canbe combined in a great number of ways, which will not enable anypractical, classification. To avoid this inconvenience, the number ofbits of each pixel is compressed by encoding as will further bedescribed later. More specifically, each pixel is compressed to a codeof one bit by the use of ADRC (encoding adaptive to the dynamic range).Thus, it will suffice to classify pixels into 16 classes (which wouldotherwise be 24 classes). Note that a code indicative of a class iscalled “index”. The number of bits of a pixel may be reduced by thevector quantization, not by the ADRC. In addition, the MSB of each pixelmay be collectively taken as the index. The classification with a codeof 4 bits, derived from normalization to one bit of 4 pixels near apredictive pixel, will lead to a classification of pixels according toan approximate pattern of a time-space change of an image.

[0350]FIG. 29 shows, by way of example, the configuration of the datacommunication system for a learning to make in the classification. Manystationary image data having different patterns should desirably besupplied to an input terminal 201 in FIG. 29. The input image data aresupplied to a time-series conversion circuit 202 and least squarecomputation circuit 203. The time-series conversion circuit 202 willsynchronize a plurality of pixel data included in for exampleraster-scanned input image data and used in the linear primarycombination model and classification.

[0351] Output data from the time-series conversion circuit 202 aresupplied to the computation circuit 203 and classification circuit 204.The classification circuit 204 will generate an index corresponding to athree-dimensional change of the image, and the index is supplied to thecomputation circuit 203. With respect to the time-series model shown inFIG. 28, the computation circuit 203 will use the aforementioned leastsquare algorithm to determine a set of coefficients wi for each classindicated with the index. The coefficients wi are stored into thecoefficient memory 205.

[0352]FIG. 30 shows, by way of example, the construction of the encoderfor the predictive coding according to the present invention.

[0353] As shown in FIG. 30, the encoder includes a coefficient memory215 having stored therein for each class coefficients having beenacquired through the above learning. Input image data from an inputterminal 211 are supplied to a time-series conversion circuit 212 whichprovides output data to a classification circuit 214 and estimated valuegeneration circuit 216. The time-series conversion circuit 212 outputspixel data supplied in the order of raster scanning for example,including pixels having been sent one after another in the past andthose which will be sent in the future, together and at a time as shownin FIG. 28. The classification circuit 214 is supplied with 4-pixel datafor use in the classification. An index from the classification circuit214 is supplied to the coefficient memory 215, and a set of coefficientscorresponding to a class is read from the coefficient memory 215.

[0354] The coefficients thus read from the coefficient memory 215 aresupplied to the estimated value generation circuit 216 where anestimated value is generated from the pixel data supplied from thetime-series conversion circuit 212. The estimated value is supplied to asubtraction circuit 218 which is also supplied with input image data viaa delay circuit 217. The time-series conversion circuit 212,classification circuit 214, coefficient memory 215 and the estimatedvalue generation circuit 216 form together a local decoder. Thesubtraction circuit 218 generates a difference between an true value(real data) and the estimated value.

[0355] The difference is encoded by an encoder 219 for encoding bycompression, and an encoded data (code) is delivered at an outputterminal 220. For the purpose of the compression-encoding, the number ofbits is reduced by the adaptive quantization, and then the data isentropy-encoded with a Huffman code predetermined based on the incidenceprobability. Further, pixel values themselves are supplied to theencoder 219 for the purpose of refreshing, and they are inserted at eachpredetermined data to be sent. However, the compression-encoding may bedone otherwise. For example, the difference value is blocked, thensubjected to an orthogonal transformation like DCT transform and furtherentropy-encoded. The compression-encoded data output is sent.

[0356] In the example shown in FIG. 30, there is used a versatilecoefficient acquired through a previously-made learning. The encodershown in FIG. 31 is designed to update the coefficient in associationwith, an input image. That is, the encoder has provided therein a leastsquare computation circuit 213 similar to that intended for the learningto provide a coefficient with which the coefficient memory 215 isupdated. More particularly, an index is supplied from the classificationcircuit 214 to the computation circuit 213 and coefficient memory 215,input image data are supplied to the computation circuit 213, and adetermined coefficient is supplied from the computation circuit 213 tothe coefficient memory 215. The coefficient may be updated based on aresult of learning considerably many frames, not only at each frame ofthe input image. Also, a coefficient determined through learning inadvance may be stored in a coefficient memory 215 in FIG. 31, and beupdated with an input image. In the example in FIG. 31, since thecontent of the coefficient memory 215 changes, it is necessary to sendcoefficient data delivered at an output terminal 221 along with a code.In the encoder shown in FIG. 31, the least square computation circuit213 may be adapted to selectively be operated.

[0357] Next, the construction of each of circuits provided duringlearning in the encoder will be described in detail.

[0358]FIG. 32 shows an example of the classification circuits 204 and214. Input image data (4-pixel data) are supplied one after another to a1-bit ADRC circuit 227. The ADRC circuit 227 detects maximum and minimumvalues of the four pixels, to thereby detect a dynamic range DR which isa difference between the maximum and minimum values. The value of eachpixel is divided by the dynamic range DR. The quotient is compared with0.5. When the quotient is larger than 0.5, a code “1” is generated. Whenthe quotient is smaller than 0.5, a code “0” is generated.

[0359] The ADRC circuit 227 compresses the value of each pixel to onebit (“0” or “1”). The output of the ADRC circuit 227 is supplied to ashift register 228 which in turn will converts the data from series toparallel. L1, L2, L3 and L4 each of 1 bit from the shift register 228are stored into a register 229. These four bits (L1, L2, L3 and L4) formtogether an index.

[0360] The ADRC circuit 227 reduces the number of quantized bits to one(bit) as in the above. However, the number of bits may be reduced to twoor more (bits). Also, the number of pixels in each pixel block includingthe four pixels for example may be reduced to a variable one.

[0361] Now, there will be described with reference to FIG. 33 acircuitry, in the receiver 2, intended for reception and decoding of thedata (received code) encoded by the classification-adaptive predictiveencoding as in the above.

[0362] As shown in FIG. 33, a received code supplied to an inputterminal 241 is supplied to a compression-encoding decoder 243, and areceived coefficient from an input terminal 242 is supplied to acoefficient memory 244. The coefficient memory 244 outputs a coefficientcorresponding to a class in responsive to the index supplied from aclassification circuit 245. When it is not intended to change anycoefficient acquired through the learning, the content of thecoefficient memory 244 will not be changed. The content of thecoefficient memory 244 has to be the same as that at the sending side.The coefficient is supplied to an estimated value generation circuit246.

[0363] The decoder 243 provides decoded difference data which will besupplied to an adder 247. Also, pixel values inserted for refreshing aresupplied to the estimated value generation circuit 246 in which they areused for generation of an estimated value. The adder 247 adds thedecoded difference value and estimated value supplied from the estimatedvalue generation circuit 246 to generate a decoded value. The decodedvalue is supplied to a time-series conversion circuits 248 and 249. Thetime-series conversion circuit 248 collects a plurality of decoded pixeldata necessary for the estimation. The time-series conversion circuit249 converts the decoded signals for display on the display unit 137 andsends the signals from an output terminal 250 to the display unit 137.

[0364] The decoded pixel data from the time-series conversion circuit248 are supplied to the estimated value generation circuit 246 and thedecoded pixel data necessary for the classification are supplied to aclassification circuit 245. The classification circuit 245 is includedin the aforementioned 1-bit ADRC circuit, and provides an index to thecoefficient memory 244. The construction of each circuit in the decodershown in FIG. 33 is similar to that having been described concerning theencoder, and therefore will not be described any more.

[0365] Next, the automatic power-consumption control algorithm will bedescribed.

[0366] For control of the power consumption, each process is dividedinto basic operations and it is preset how much power each of theoperations consumes. The value of the power for each operation will behardware-dependent. With the use of the value of power for eachoperation, the power consumption can be reduced. In this embodiment,charging is made for usage of the automatic power-consumption controlalgorithm.

[0367] Also, the power consumption should automatically be controlled aselaborately as possible. Desirably, the power-consumption control shouldbe effected for each process (1 clock) in the circuit. The concept ofthe power consumption control is shown in FIG. 34.

[0368] In FIG. 34, the vertical axis indicates a power consumption whilethe horizontal axis indicates one clock (one process). As shown in FIG.34, the minimum necessary power for each of very small processes isexamined in advance to control the supply of a necessary power for aprocess going to be done. That is, the power control is made to decreasethe power supply for a process needing not so much power whileincreasing the power supply for a process needing much power.

[0369]FIG. 35 shows a flow of operations made in judging in theautomatic power-consumption control algorithm whether the power controlshould be done for each of processes (other than the power controlprocess).

[0370] As in FIG. 35, the power consumption controller 131 in FIG. 21judges in step S151 based on the automatic power-consumption controlalgorithm which process is going to be lone. When the power consumptioncontroller 131 judges that the process going to be done is a one whichuses the automatic power-consumption control algorithm, it will is setto perform the power control function for that process in step S152.When the judgement is that the process is a one which does not use theautomatic power-consumption control algorithm, the power consumptioncontroller 131 goes to step S 153 where it will be set not to performthe power control function.

[0371] Thereafter, the power consumption controller 131 goes to step S154 where it will judge whether it should exit the power controlfunction. When the power consumption controller 131 should not end thefunction, it returns to step S151. When it should end the function, itexits the automatic power-consumption control algorithm.

[0372]FIG. 36 shows a flow of operations effected when the powerconsumption controller 131 is set in step S152 in FIG. 35 to perform thepower control function.

[0373] In step S161 in FIG. 36, the power consumption controller 131monitors a next process to make, and computes, in step S162, a powernecessary for the process (it is necessary to supply an power amountpreset based on a reference table).

[0374] Next in step S 163, the power consumption controller 131supplies, from a power source (not shown), a power computed as in theabove to each circuit in the system. Thus in step S164, each circuit inthe receiver 2 actually effects the above-mentioned process.Alternatively, in case the power consumption controller 131 isincorporated in the controller 133 or the like, the system may beadapted such that some circuits in the controller 133 are selectivelymade to sleep according to the operation type of each process in orderto control the power consumption.

[0375] Thereafter, the power consumption controller 131 judges whetherthe power supply is to be ended. When the judgment is that the powersupply is not to be ended, the power consumption controller 131 returnsto step S161. If the power consumption controller 131 judges that thepower supply should be ended, it will exit the automaticpower-consumption control algorithm.

[0376] Note that the charged amount may be changed according to thelevel of the process (algorithm). For example, the charged amount may bechanged according to the time intervals of monitoring, to the level ofoptimization of the power consumption control, or to the like.

[0377] The above series of operations can be done by hardware orsoftware. In case the operations are to be done by software, programsforming together the software are installed in a computer incorporatedin the transmitter 1 and receiver 2 as dedicated hardware, agenera-purpose computer, or the like.

[0378] Here will be described a recording medium having the program fordoing the series of operations recorded therein and used to install theprogram into a computer and enable them to be executed by the computer.

[0379] The program for effecting the above series of operations can bepre-recorded in a hard disc or semiconductor memory as a recordingmedium incorporated in a computer. The program may be stored (recorded)provisionally or permanently in recording medium such as a floppy disc,CD-ROM (compact disc read-only memory), MO (magneto-optical) disc, DVD(digital versatile disc), magnetic disc or a semiconductor memory.

[0380] Note that the program can be installed from the above-mentionedrecording medium to a computer or it can be wirelessly transferred froma download site to a computer via a digital-broadcasting artificialsatellite or via a network such as LAN (local area network) or Internetand installed into a hard disc incorporated in the computer.

[0381] Also note that the steps for description of the program forvarious operations may not always be done on the time series in adescription sequence in a flow chart but may include operations whichare effected in parallel or individually (for example, paralleloperations or object-based operations).

[0382] Next, an example construction of the above-mentioned computerwill be described with reference to FIG. 37.

[0383] The computer shown in FIG. 37 incorporates a CPU (centralprocessing unit) 142. An input/output interface 145 is connected via abus 141 to the CPU 142. When supplied with an instruction from the useroperating an input unit 147 including a keyboard, mouse and the like viathe input/output interface 145, the CPU 142 will execute a programstored in a ROM (read-only memory) 143 corresponding to theabove-mentioned semiconductor memory. Alternatively, the CPU 142 willload, into a RAM (random-access memory) 144, and execute, a programstored in the hard disc 140, a program transferred from the satellite ornetwork, received by a communication unit 148 and installed in the harddisc 140 or a program read out from a floppy disc, CD-ROM, MO disc, DVDor magnetic disc provided in a drive 149 and installed in the hard disc140. Then the CPU 142 outputs the result of execution of the program toa display unit 146 composed of an LCD (liquid crystal display) or thelike via the input/output interface 145 for example.

[0384] According to the present invention, the time resolution andspatial resolution of an image can be changed also by making discretecosine conversion of the image to select a degree of coefficient orquantizing the image to change the quantizing step in the transmitter 1for example.

[0385] For displaying for example an object image (interesting area) atan ordinary time resolution, the profile of the image may bechain-encoded, a mean value of pixels (color) of the object image bedetermined as a representative value and the data be subject to anentropy encoding such as Huffman encoding, in the transmitter 1, whilethe inside of the object image area may be painted in a color as therepresentative value in the receiver 2.

[0386] In the above example, the spatial resolution of an image isimproved. On the contrary, the time resolution of the image can beimproved. Also in the above example, the spatial resolution of apreferred range as an area of an image is improved, but the spatialresolution of the entire image can also be improved.

[0387] Further, in the above example, an image is separated into abackground and object before being processed. However, the image may beprocessed without being so separated.

[0388] In the above example, switching can be made between various kindsof hardware by selecting an algorithm or software layer. In the presentinvention, the algorithm and hardware control program include anoperation for switching between pieces of hardware. Thus, the presentinvention will include the switching between pieces of hardware and alsocharging corresponding to the switching.

[0389] Besides, the present invention is applicable to image data aswell as to sound data. For example, the present invention can be appliedto extraction of sound features (such as sound pitch, desired part ofhuman voice, characteristic sounds of musical instruments, etc.) basedon a certain basic frequency included in sound signal.

[0390] As having previously been described, the Japanese PublishedUnexamined Application No. 164552 of 1998 discloses a video-on-demandtransmitter and terminal which make it possible to serve a video programwhose quality meets the user's demand and charge the user on aprogram-quality basis. In the technique disclosed in this JapanesePublished Unexamined Application, the transmitter and terminal use dataprocessing algorithms similar to each other such as MPEG-1 or MPEG-2upon which the quality of a served video program depends and thus thecharge for a served video program is varied depending upon whether ornot the served video program is a one compressed with a data processingalgorithm such as MPEG-1 or MPEG-2. That is, in this system, a dataprocessing algorithm (MPEG-1 or MPEG-2) used at the receiving side(terminal) is similar to a one (MPEG-1 or MPEG-2) at the transmitter toselect an image quality. Also in this disclosed system, it is possibleat the receiving side to reproduce an video program having beenprocessed with either MPEG-1 or MPEG-2. In other words, the sending sideuses a data processing algorithm similar to a one used at the receivingside.

[0391] However, in the data transmission system according to the presentinvention selects, the transmitter selects one of quite differentalgorithms such as sub sample compression, object encoding-compression,classification-adaptive predictive compression, etc. for transmission ofdata and the receiver can reproduce the data having been processed withthe selected algorithm. Further, the communication system according tothe present invention cannot only select one of the different algorithmsbut can automatically select a compression method and predict a focus ofinterest. Also the system according to the present invention can selectone of different software layers as in the automatic selective controlof power consumption.

INDUSTRIAL APPLICABILITY

[0392] According to the present invention, desired ones of a pluralityof different software layers and algorithms can be designated for use toprocess data for transmission, ones for the data can be selected fromthe different software layers and algorithms, the data can be processedwith the use of the selected software and algorithm, and charging can bemade for the use of the software layer and algorithm, whereby it ispossible to serve data to various demands from a user and charge theuser for the use of the data, that is, to charge the user for the meritssuch as level of data processing result (image quality, sound quality,etc.), power consumption, processing time, data transfer time and resultof processing as well as to the satisfaction of the data server orprovider having served the data to the user.

1. A data processing system comprising: a designation input means fordesignating desired methods for at least an application layer andhardware control layer, respectively; a signal processing means forselectively carrying out methods related to the application layer; ahardware controlling means for selectively carrying out methods relatedto the hardware control layer; and means for controlling the signalprocessing means or hardware controlling means based on the designationfrom the designation input means to carry out the designated method. 2.The data processing system according to claim 1, wherein: the signalprocessing means includes a storage means to store a plurality ofmethods; and the controlling means reads, from the storage means, onemethod in response to a designation from the designation input means andcarries out the method.
 3. The data processing system according to claim1, further comprising: means for sending data; means for receivingtransmitted data; and means for outputting received content data; andthe signal processing means being controlled by the controlling means toprocess supplied content data by carrying out a method selected on thebasis of the designation supplied from the designation input means, andoutput the processed content data to the transmitting means.
 4. The dataprocessing system according to claim 3, wherein the signal processingmeans has a plurality of methods corresponding to a plurality ofcompression algorithms by which the supplied content data arecompressed; and the content data are compressed by a user-definedcompression algorithm under the control of the controlling means.
 5. Thedata processing system according to claim 4, wherein the signalprocessing means holds a method which is a compression algorithmincluding the steps of: extracting an interesting image blockcorresponding to an interesting pixel from image data supplied ascontent data; classifying the interesting pixel based on the values ofpixels included in the interesting image block; outputting an estimatedvalue of the interesting pixel, obtained by a linear primary combinationof a plurality of coefficients pre-computed for each class and storedand the value of each pixel included in the estimation image blockcorresponding to the interesting pixel; and encoding a value resultedfrom subtraction of the estimated value from the value of theinteresting pixel.
 6. The data processing system according to claim 4,wherein the signal processing means holds a method which is acompression algorithm including the steps of: dividing a current frameof moving image data supplied as content data into more than at leastone object images; and computing a movement vector of an object insidethe object image resulted from the division from the current frame andpast frame.
 7. The data processing system according to claim 4, whereinthe signal processing means holds a method which is a compressionalgorithm to automatically select, from the signal processing means, oneof a plurality of methods corresponding to the plurality of compressionalgorithms correspondingly to money amount information designated by thedesignation input means.
 8. The data processing system according toclaim 4, wherein the method to select one of the plurality of methodsincludes the steps of: selecting one, which can be used at the lowestprice, of the plurality of methods corresponding to the plurality ofcompression algorithms; computing a difference between decoded dataresulted from decoding of the content data compressed by thelowest-price method and original content data; and outputting thecompressed content data when the difference is smaller than a threshold.9. The data processing system according to claim 4, wherein the signalprocessing means holds a method to select, from the signal processingmeans, one of a plurality of methods corresponding to the plurality ofcompression algorithms correspondingly to a feature of image data as thesupplied content data.
 10. The data processing system according to claim9, wherein the method held by the signal processing means to select oneof the plurality of methods includes the steps of: detecting an activityas a feature of image data as the supplied content data; and selecting aone, of the plurality of methods, corresponding to a higher-levelcompression algorithm.
 11. The data processing system according to claim4, wherein: the output means displays image data as the supplied contentdata; the designation input means is operated by the user to designate apredetermined position of the displayed image data; and the signalprocessing means holds a method including the steps of: estimating thepreference of the user from a feature of an object image in the imagedata corresponding to a predetermined position designated by thedesignation input means; automatically extracting an object image theuser is interested in from the image data based on the estimated user'spreference; and compressing the image data for the extracted objectimage to have a lower compression ratio than other object images. 12.The data processing system according to claim 7, further comprising: ausage charge storage means for storing information on a charge for theuse of each of the plurality of methods stored in the signal processingmeans or hardware controlling means; and means for computing informationindicative of an amount charged to the user having given a command tothe designation input means and outputting the charged amountinformation.
 13. The data processing system according to claim 12,wherein the computing means computes the charged amount information bysumming a charge for the use of the automatic selection method forautomatic selection of one of the plurality of methods based on theinformation stored in the usage charge storage means and a charge forthe use of a compression algorithm selected by the automatic selectionmethod.
 14. The data processing system according to claim 12, whereinthe computing means sends a pre-stored user's number of accounting, anumber of accounting of a provider providing a service based on themethod and the charged amount information to an external payment center.15. The data processing system according to claim 1, wherein thehardware controlling means holds a method to control a power for supplyto at least the controlling means according to a command supplied perclock.
 16. The data processing system according to claim 15, wherein:the hardware controlling and controlling means are formed in one chip;and the method of the hardware controlling means puts the functions ofthe controlling means into sleeping state according to a commandsupplied per clock.
 17. The data processing system according to claim16, further comprising: a usage charge storage means for storinginformation on a charge for the use of each of the plurality of methodsstored in the signal processing means or hardware controlling means; andmeans for computing information indicative of an amount charged to theuser having given a command to the designation input means andoutputting the charged amount information.
 18. The data processingsystem according to claim 17, wherein the computing means sends apre-stored user's number of accounting, a number of accounting of aprovider providing a service based on the method and the charged amountinformation to an external payment center.
 19. The data processingsystem according to claim 2, wherein: the signal processing means islocated remote from the controlling means; the one method designatedfrom the designation input means is read from the storage means anddownloaded to the controlling means; and the controlling means carriesout the downloaded method.
 20. A communication system with a chargingmanagement apparatus, comprising: a processor including: a designationinput means for designating desired methods for at least an applicationlayer and hardware control layer, respectively; a request signaltransmitting means for sending a signal of request for a methoddesignated by the designation input means; means for receiving a replyto the request signal; a signal processing means for selectivelycarrying out methods related to the application layer; a hardwarecontrolling means for selectively carrying out methods related to thehardware control layer; and means for controlling the signal processingmeans or hardware controlling means to carry out a method designated bythe designation input means; a charging management apparatus including:means for receiving the request signal; for judging, based on thereceived request signal, whether the method is available not; means forsending the judgment result from the judging means to the receivingmeans; a usage-charge storage means for storing charge information onthe use of each of a plurality of methods stored in the signalprocessing means or hardware controlling means; and means for computing,based on information stored in the usage-charge storage means,information indicative of an amount charged to the user having madedesignation to the designation input means, and outputting the chargedamount information.
 21. The communication system according to claim 20,the signal processing means includes a storage means to store aplurality of methods; and the controlling means reads, from the storagemeans, one method in response to a designation from the designationinput means and carries out the method.
 22. The communication systemaccording to claim 20, wherein: the receiving means receives contentdata; the processor includes an output means for outputting the receivedcontent data; and the signal processing means being controlled by thecontrolling means to process supplied content data by carrying out amethod selected on the basis of the designation supplied from thedesignation input means, and output the processed content data to thetransmitting means.
 23. The communication system according to claim 22,wherein the signal processing means has a plurality of methodscorresponding to a plurality of compression algorithms by which thesupplied content data are compressed; and the content data arecompressed by a user-defined compression algorithm under the control ofthe controlling means.
 24. The communication system according to claim23, wherein the signal processing means holds a method which is acompression algorithm including the steps of: extracting an interestingimage block corresponding to an interesting pixel from image datasupplied as content data; classifying the interesting pixel based on thevalues of pixels included in the interesting image block; outputting anestimated value of the interesting pixel, obtained by a linear primarycombination of a plurality of coefficients pre-computed for each classand stored and the value of each pixel included in the estimation imageblock corresponding to the interesting pixel; and encoding a valueresulted from subtraction of the estimated value from the value of theinteresting pixel.
 25. The communication system according to claim 23,wherein the signal processing means holds a method which is acompression algorithm including the steps of: dividing a current frameof moving image data supplied as content data into more than at leastone object images; and computing a movement vector of an object insidethe object image resulted from the division from the current frame andpast frame.
 26. The communication system according to claim 23, whereinthe signal processing means holds a method which is a compressionalgorithm to automatically select, from the signal processing means, oneof a plurality of methods corresponding to the plurality of compressionalgorithms correspondingly to money amount information designated by thedesignation input means.
 27. The communication system according to claim26, wherein the method to select one of the plurality of methods has thesteps of: selecting one, which can be used at the lowest price, of theplurality of methods corresponding to the plurality of compressionalgorithms; computing a difference between decoded data resulted fromdecoding of the content data compressed by the lowest-price method andoriginal content data; and outputting the compressed content data whenthe difference is smaller than a threshold.
 28. The communication systemaccording to claim 23, wherein the signal processing means holds amethod to automatically select, from the signal processing means, one ofa plurality of methods corresponding to the plurality of compressionalgorithms correspondingly to a feature of image data as the suppliedcontent data.
 29. The communication system according to claim 28,wherein the method held by the signal processing means to select one ofthe plurality of methods is to detect an activity as a feature of imagedata as the supplied content data, and select a one, of the plurality ofmethods, corresponding to a higher-level compression algorithm.
 30. Thecommunication system according to claim 23, wherein: the output meansdisplays image data as the supplied content data; the designation inputmeans is operated by the user to designate a predetermined position ofthe displayed image data; and the signal processing means holds a methodincluding the steps of: estimating the preference of the user from afeature of an object image in the image data corresponding to apredetermined position designated by the designation input means;automatically extract an object image the user is interested in from theimage data based on the estimated user's preference; and compressing theimage data for the extracted object image to have a lower compressionratio than other object images.
 31. The data processing system accordingto claim 20, wherein the computing means computes the charged amountinformation by summing a charge for the use of the automatic selectionmethod for automatic selection of one of the plurality of methods basedon the information stored in the usage charge storage means and a chargefor the use of a compression algorithm selected by the automaticselection method.
 32. The communication system according to claim 20,wherein the computing means sends a pre-stored user's number ofaccounting, a number of accounting of a provider providing a servicebased on the method and the charged amount information to an externalpayment center.
 33. The communication system according to claim 20,wherein the hardware controlling means holds a method to control a powerfor supply to at least the controlling means according to a commandsupplied per clock.
 34. The communication system according to claim 33,wherein: the hardware controlling and controlling means are formed inone chip; and the method of the hardware controlling means puts thefunctions of the controlling means into sleeping state according to acommand supplied per clock.
 35. The communication system according toclaim 34, wherein the computing means sends a pre-stored user's numberof accounting, a number of accounting of a provider providing a servicebased on the method and the charged amount information to an externalpayment center.
 36. The communication system according to claim 21,wherein: the signal processing means is located remote from thecontrolling means; the one method designated from the designation inputmeans is read from the storage means and downloaded to the controllingmeans; and the controlling means carries out the downloaded method. 37.A charging management apparatus comprising: means for receiving, from anexternal apparatus, a signal of request for permission to use desiredmethods related to an application layer and hardware control layer,respectively, of the external apparatus; means for judging, based on thepermission request signal received by the request signal receivingmeans, whether the method is available not; means for sending thejudgment result from the judging means to the receiving means; ausage-charge storage means for storing charge information on the use ofeach of a plurality of methods; and means for computing, based oninformation stored in the usage-charge storage means, informationindicative of an amount charged to the user having made designation tothe designation input means, and outputting the charged amountinformation.
 38. The charging management apparatus according to claim37, wherein the charged amount information computing means computes,based on information stored in the usage-charge storage means, chargedamount information by summing a charge for use of an automatic selectionmethod for automatically selecting one of a plurality of methods and acharge for the use of a method corresponding to a compression algorithmselected by the automatic selection method.
 39. The charging managementapparatus according to claim 37, wherein the computing means sends apre-stored user's number of accounting, a number of accounting of aprovider providing a service based on the method and the charged amountinformation to an external payment center.
 40. A communication systemcomprising first and second communication apparatuses and a chargingapparatus, the first communication apparatus including: a designationinput means for designating desired methods for at least an applicationlayer and hardware control layer, respectively, of a transmitter; meansfor sending a signal of request for a method designated by thedesignation input means; means for receiving content data processed bythe method made available by the request signal transmitting means; ahardware controlling means for selectively carrying out methods relatedto the hardware control layer; and a first controlling means forcontrolling the hardware controlling means to carry out a methoddesignated by the designation input means; the second communicationapparatus including: a signal processing means for selectively carryingout methods related to the application layer to process input contentdata; a second controlling means for controlling the method designatedby the designation input means to be carried out based on the requestsignal sent from the request signal transmitting means; and means forsending the processed content data to outside; and the chargingmanagement apparatus including: a usage-charge storage means for storingcharge information on the use of each of a plurality of methods storedin the signal processing means or hardware controlling means; and meansfor computing, based on information stored in the usage-charge storagemeans, information indicative of an amount charged to the user havingmade designation to the designation input means, and outputting thecharged amount information.
 41. The communication system according toclaim 40, wherein: the signal processing means includes a storage meansto store a plurality of methods; and the first and second controllingmeans read, from the storage means, one method in response to adesignation from the designation input means and carries out the method.42. The communication system according to claim 41, wherein the signalprocessing means holds a plurality of methods corresponding to aplurality of compression algorithms by which the supplied content dataare compressed; and the content data are compressed by a user-definedcompression algorithm under the control of the second controlling means.43. The communication system according to claim 41, wherein the signalprocessing means holds a method which is a compression algorithm havingthe steps of: extracting an interesting image block corresponding to aninteresting pixel from image data supplied as content data; classifyingthe interesting pixel based on the values of pixels included in theinteresting image block; outputting an estimated value of theinteresting pixel, obtained by a linear primary combination of aplurality of coefficients pre-computed for each class and stored and thevalue of each pixel included in the estimation image block correspondingto the interesting pixel; and encoding a value resulted from subtractionof the estimated value from the value of the interesting pixel.
 44. Thecommunication system according to claim 42, wherein the signalprocessing means holds a method which is a compression algorithmincluding the steps of: dividing a current frame of moving image datasupplied as content data into more than at least one object images; andcomputing a movement vector of an object inside the object imageresulted from the division from the current frame and past frame. 45.The communication system according to claim 42, wherein the signalprocessing means holds a method which is a compression algorithm toautomatically select, from the signal processing means, one of aplurality of methods corresponding to the plurality of compressionalgorithms correspondingly to money amount information designated by thedesignation input means.
 46. The communication system according to claim45, wherein the method to select one of the plurality of methodsincludes the steps of: selecting one, which can be used at the lowestprice, of the plurality of methods corresponding to the plurality ofcompression algorithms; computing a difference between decoded dataresulted from decoding of the content data compressed by thelowest-price method and original content data; and outputting thecompressed content data when the difference is smaller than a threshold.47. The communication system according to claim 42, wherein the signalprocessing means holds a method to automatically select, from the signalprocessing means, one of a plurality of methods corresponding to theplurality of compression algorithms correspondingly to a feature ofimage data as the supplied content data.
 48. The communication systemaccording to claim 47, wherein the method held by the signal processingmeans to select one of the plurality of methods has the steps of:detecting an activity as a feature of image data as the supplied contentdata; and selecting a one, of the plurality of methods, corresponding toa higher-level compression algorithm.
 49. The communication systemaccording to claim 42, wherein: there is further provided a displaymeans for displaying image data as the supplied content data; thedesignation input means is operated by the user to designate apredetermined position of the displayed image data; and the signalprocessing means holds a method including the steps of: estimating thepreference of the user from a feature of an object image in the imagedata corresponding to a predetermined position designated by thedesignation input means; automatically extracting an object image theuser is interested in from the image data based on the estimated user'spreference; and compressing the image data for the extracted objectimage to have a lower compression ratio than other object images. 50.The communication system according to claim 45, wherein the computingmeans computes the charged amount information by summing a charge forthe use of the automatic selection method for automatic selection of oneof the plurality of methods based on the information stored in the usagecharge storage means and a charge for the use of a compression algorithmselected by the automatic selection method.
 51. The communication systemaccording to claim 50, wherein the computing means sends a pre-storeduser's number of accounting, a number of accounting of a providerproviding a service based on the method and the charged amountinformation to an external payment center.
 52. The data processingsystem according to claim 40, wherein the hardware controlling meansholds a method to control a power for supply to at least the controllingmeans according to a command supplied per clock.
 53. The communicationsystem according to claim 52, wherein: the hardware controlling andcontrolling means are formed in one chip; and the method of the hardwarecontrolling means puts the functions of the controlling means intosleeping state according to a command supplied per clock.
 54. Thecommunication system according to claim 53, wherein the computing meanssends a pre-stored user's number of accounting, a number of accountingof a provider providing a service based on the method and the chargedamount information to an external payment center.
 55. A communicationapparatus comprising: a designation input means for designating desiredmethods to at least an application layer and hardware control layer,respectively; means for sending a signal of request for the methoddesignated by the designation input means; means for receiving contentdata processed by a method corresponding to the request signaltransmission means; a hardware controlling means for selectivelycarrying out methods related to the hardware control layer; and meansfor controlling the hardware controlling means based on the designationfrom the designation input means to carry out the designated method. 56.The communication apparatus according to claim 55, wherein the hardwarecontrolling means holds a method to control the power supply to at leastthe controlling means according to a command supplied per clock.
 57. Thecommunication apparatus according to claim 56, wherein: the hardwarecontrolling means are formed in one chip; and the method of the hardwarecontrolling means puts the functions of the controlling means intosleeping state according to a command supplied per clock.
 58. Thecommunication apparatus according to claim 55, wherein the receivingmeans receives a billing signal for a usage charge based on usage chargeinformation for each of a plurality of methods stored in the hardwarecontrolling means.
 59. A communication apparatus for communication ofcontent data, comprising: means for receiving a signal of request forcarrying out a method in an external application layer; a signalprocessing means for processing supplied content data by selectingcarrying out a method related to the application layer; means forcontrolling the signal processing means to carry out the receivedrequest signal to carry out the designated method; and means for sendingthe processed content data to outside.
 60. The communication apparatusaccording to claim 59, wherein: the signal processing means includes astorage means to store a plurality of methods; and the controlling meansreads, from the storage means, one method in response to the requestsignal and carries out the method.
 61. The communication apparatusaccording to claim 60, wherein the signal processing means holds aplurality of methods corresponding to a plurality of compressionalgorithms by which the supplied content data are compressed; and thecontent data are compressed by a user-defined compression algorithmunder the control of the controlling means.
 62. The communicationapparatus according to claim 60, wherein the signal processing means hasa method which is a compression algorithm including the steps of:extracting an interesting image block corresponding to an interestingpixel from image data supplied as content data; classifying theinteresting pixel based on the values of pixels included in theinteresting image block; outputting an estimated value of theinteresting pixel, obtained by a linear primary combination of aplurality of coefficients pre-computed for each class and stored and thevalue of each pixel included in the estimation image block correspondingto the interesting pixel; and encoding a value resulted from subtractionof the estimated value from the value of the interesting pixel.
 63. Thecommunication apparatus according to claim 61, wherein the signalprocessing means holds a method which is a compression algorithmincluding the steps of: dividing a current frame of moving image datasupplied as content data into more than at least one object images; andcomputing a movement vector of an object inside the object imageresulted from the division from the current frame and past frame. 64.The communication apparatus according to claim 61, wherein the signalprocessing means holds a method which is a compression algorithm toautomatically select, from the signal processing means, one of aplurality of methods corresponding to the plurality of compressionalgorithms correspondingly to money amount information designated withthe request signal.
 65. The communication system according to claim 64,wherein the method to select one of the plurality of methods has thesteps of: selecting a one, which can be used at the lowest price, of theplurality of methods corresponding to the plurality of compressionalgorithms; computing a difference between decoded data resulted fromdecoding of the content data compressed by the lowest-price method andoriginal content data; and outputting the compressed content data whenthe difference is smaller than a threshold.
 66. The communicationapparatus according to claim 61, wherein the signal processing meansholds a method to select, from the signal processing means, one of aplurality of methods corresponding to the plurality of compressionalgorithms correspondingly to a feature of image data as the suppliedcontent data.
 67. The communication apparatus according to claim 66,wherein the method held by the signal processing means to select one ofthe plurality of methods has the steps of: detecting an activity as afeature of image data as the supplied content data; and selecting a one,of the plurality of methods, corresponding to a higher-level compressionalgorithm.
 68. The communication apparatus according to claim 61,wherein the signal processing means holds a method including the stepsof: estimating the preference of the user from a feature of an objectimage in the image data corresponding to a predetermined positiondesignated by the designation input means; automatically extracting anobject image the user is interested in from the image data based on theestimated user's preference; and compressing the image data for theextracted object image to have a lower compression ratio than otherobject images.
 69. The communication system according to claim 64,further comprising: a usage charge storage means for storing informationon a charge for the use of each of the plurality of methods stored inthe signal processing means or hardware controlling means; and means forcomputing information indicative of an amount charged to the user havinggiven a command to the designation input means and outputting thecharged amount information.
 70. The communication apparatus according toclaim 69, wherein the computing means computes the charged amountinformation by summing a charge for the use of the automatic selectionmethod for automatic selection of one of the plurality of methods basedon the information stored in the usage charge storage means and a chargefor the use of a compression algorithm selected by the automaticselection method.
 71. The communication apparatus according to claim 70,wherein the computing means sends a pre-stored user's number ofaccounting, a number of accounting of a provider providing a servicebased on the method and the charged amount information to an externalpayment center.
 72. A data processing method comprising the steps of:designating desired methods for at least an application layer andhardware control layer, respectively; carrying out the designated methodfor the application layer; and carrying out the designated method forthe hardware control layer.
 73. A communication method for computinginformation an amount charged to the user, comprising the steps of:designating desired methods for at least an application layer andhardware control layer, respectively; judging whether the designatedmethods are available or not; carrying out, by a signal processing meansor hardware controlling means, the requested method according to thejudgment that the method is available or not; and computing informationon an amount charged to the user having designated the method based oninformation on usage charge for the method.
 74. A charging method forpresenting charging information to the user, comprising the steps of:receiving, from an external apparatus, a signal of request forpermission of the use of desired methods for an application layer andhardware control layer, respectively, of the external apparatus; judgingwhether the methods are available or not, based on the received requestsignal; and computing, correspondingly to the result of judgment,information on an amount charged to the user having sent the requestsignal based on usage charge information for each of the methods, andoutputting the charged amount information.
 75. A communication methodfor communication of content data, comprising the steps of: designatingdesired methods for at least an application layer of a transmitter and ahardware control layer, respectively, of a receiver; sending a signal ofrequest for the designated method; processing content data input to thetransmitter by a signal processing means by carrying out the methodaccording to the sent request signal; sending the processed content datafrom the transmitter to the receiver; receiving the sent content data bythe receiver; controlling the power supply to an internal circuit of thereceiver by a method carried out by a hardware controlling meanscorrespondingly to the designated method; and computing information onan amount charged to the user having designated the methods based oninformation stored in a usage charge information storage means whichstores usage charge information for each of a plurality of methods heldin the signal processing means or hardware controlling means, andoutputting the charged amount information.
 76. A communication methodfor communication of content data, comprising the steps of: designatingdesired methods for at least an application layer and hardware controllayer, respectively; sending a signal of request for the designatedmethod; receiving content data processed by the method corresponding tothe request signal; and carrying out the designated method by a hardwarecontrolling means to control the power supply in an internal circuit ofan apparatus adopting this method.
 77. A communication method forsending content data to outside, comprising the steps of: receiving anexternal signal of request for carrying out a method for an applicationlayer; selectively carrying out, according to the received requestsignal, methods related to the application layer to process suppliedcontent data; and sending the processed content data to outside.
 78. Astorage medium having stored therein a program readable by aninformation processing means, the program comprising the steps of:designating desired methods for at least an application layer andhardware control layer, respectively; carrying out the designated methodfor the application layer; and carrying out the designated method forthe hardware control layer.
 79. A storage medium having stored therein aprogram readable by an information processing means, the programcomprising the steps of: designating desired methods for at least anapplication layer and hardware control layer, respectively; judgingwhether the designated methods are available or not; carrying out, by asignal processing means or hardware controlling means, the requestedmethod according to the judgment that the method is available or not;and computing information on an amount charged to the user havingdesignated the method based on information on usage charge for themethod.
 80. A storage medium having stored therein a program readable byan information processing means, the program comprising the steps of:receiving, from an external apparatus, a signal of request forpermission of the use of desired methods for an application layer andhardware control layer, respectively, of the external apparatus; judgingwhether the methods are available or not, based on the received requestsignal; and computing, correspondingly to the result of judgment,information on an amount charged to the user having sent the requestsignal based on usage charge information for each of the methods, andoutputting the charged amount information.
 81. A storage medium havingstored therein a program readable by an information processing means,the program comprising the steps of: designating desired methods for atleast an application layer of a transmitter and a hardware controllayer, respectively, of a receiver; sending a signal of request for thedesignated method; processing content data input to the transmitter by asignal processing means by carrying out the method according to the sentrequest signal; sending the processed content data from the transmitterto the receiver; receiving the sent content data by the receiver;controlling the power supply to an internal circuit of the receiver by amethod carried out by a hardware controlling means correspondingly tothe designated method; and computing information on an amount charged tothe user having designated the methods based on information stored in ausage charge information storage means which stores usage chargeinformation for each of a plurality of methods held in the signalprocessing means or hardware controlling means, and outputting thecharged amount information.
 82. A storage medium having stored therein aprogram readable by an information processing means, the programcomprising the steps of: designating desired methods for at least anapplication layer and hardware control layer, respectively; sending asignal of request for the designated method; receiving content dataprocessed by the method corresponding to the request signal; andcarrying out the designated method by a hardware controlling means tocontrol the power supply in an internal circuit of an apparatus adoptingthis method.
 83. A storage medium having stored therein a programreadable by an information processing means, the program comprising thesteps of: receiving an external signal of request for carrying out amethod for an application layer; selectively carrying out, according tothe received request signal, methods related to the application layer toprocess supplied content data; and sending the processed content data tooutside.